Background Replacing Apparatus, Background Replacing Program, and Background Replacing Method

ABSTRACT

There are provided a background replacing apparatus, a background replacing program, and a background replacing method, which are capable of creating a background replaced image in which a background of a photographic image is naturally replaced with a desired background. The background replacing apparatus has an image obtaining section  710  that obtains a plurality of photographic images obtained through photography of a common subject under a plurality of photographic conditions, and a replacement processing section  730 . In the replacement processing section, a background area of the photographic image is discriminated from other areas in accordance with the photographic images obtained by the image obtaining section  710 , and a mixing ratio of a subject color and a background color, and the subject color are presumed in other areas excepting the background area in the photographic image, and a background of the photographic image obtained by the image obtaining section  710  is replaced with a different background using the subject color and the mixing ratio, which are presumed.

TECHNICAL FIELD

The present invention relates to a background replacing apparatus forreplacing a background of a photographic image with another backgroundto create a background replaced image, a background replacing programthat causes a computer to operate as the background replacing apparatus,when the background replacing program is incorporated into the computerand is executed, and a background replacing method of replacing abackground of a photographic image with another background to create abackground replaced image.

BACKGROUND ART

Hitherto, photo studio and the like perform a photographic work using abackground curtain. However, for such a photographic work, there is useda large background curtain to provide a sufficient background, and thereis a need to prepare a wide photographic working space. Further,exchanges of the background curtain take a great deal of time, and as aresult, it involves a high cost of photographic work.

Thus, in order that a person can enjoy photography with ease, it isdesired to develop a small type of photographic system without need of alarge background curtain and a wide photographic working space, asneeded for the conventional photographic work. According to such a smalltype of photographic system, there is performed a photographic workbefore some provisional background instead of the background curtain asmentioned above. Therefore, there is considered an application ofbackground replacing processing, in which a background of a photographicimage is replaced with a background desired by a custom, to thephotographic image obtained through photography.

As an example of technology as to the background replacing processing,for example, there is known processing referred to as chromakeyprocessing in which photography is carried out through an arrangement ofbackground panel coated with blue and the like back the subject, and ina photographic image obtained through the photography, a portion of thesame color as the background panel and a portion of the different colorfrom the background panel are distinguished from one another in form ofa background area of the photographic image and a subject area,respectively, so that the portion, which is discriminated as thebackground area of the photographic image, is replaced with a desiredbackground.

According to the chromakey processing, in the event that a shadow of thesubject of the photographic image is formed on the background panel, itwould happen that the shadow disturbs a distinction between the subjectarea and the background area. In view of the foregoing, in the eventthat a photograph of an image of interest of the chromakey processing istaken, there is proposed technology that as the background, a backgroundpanel of a panel light emission is used, so that a photography iscarried out in a state that the shadow of the subject on the backgroundpanel is negated by light emitted from the background panel (forexample, cf. Japanese Patent Application Laid Open Gazette Tokukai2000-224410 (pages 3-10, FIG. 3)). Japanese Patent Application Laid OpenGazette Tokukai 2000-224410 further discloses chromakey processingwherein when a portion, which is discriminated as a subject area, issynthesized with a desired background, a color tone of the portiondiscriminated as the subject area is controlled to a color toneaccording to the color tone of the desired background, so that there iscreated a composite image in which the subject looks as it naturallymatches for the desired background.

Further, there is proposed technology that two photographic images areobtained through performing photography under two lighting conditions ofa state in which a lighting directed to the subject turns on at a placewhere the back of the subject opens, for example, out of doors, and astate in which the lighting turns off, and in those two photographicimages, a portion that is different in luminance and a portion that issame in luminance are distinguished as a subject area and a backgroundarea of the photographic images, respectively, so that the portiondiscriminated as the background area of the photographic images isreplaced with a desired background (for example, cf. Japanese PatentApplication Laid Open Gazette Tokukai Hei 10-210340 (pages 2-3, FIG.1)).

By the way, for example, in the event that photography is carried outtaking a person as the subject, it often happens that the photography iscarried out in a state that a background appears through hairs andscarves of the woman as the subject. Thus, it often happens that thebackground appears through a part of the subject on the photographicimage. In the event that as to the photographic image as mentionedabove, the background is replaced in accordance with technologiesdisclosed in Japanese Patent Application Laid Open Gazette Tokukai2000-224410 (pages 3-10, FIG. 3) and Japanese Patent Application LaidOpen Gazette Tokukai Hei 10-210340 (pages 2-3, FIG. 1), and generalchromakey processing, it often happens that there is created unnaturalbackground-replaced images such as an image in which a part of the edgeof the subject together with the background is replaced by anotherbackground, and an image in which the old background before replacementappears through a part of the subject.

DISCLOSURE OF THE INVENTION

In view of the foregoing, it is an object of the present invention toprovide a background replacing apparatus capable of naturally replacinga background of a photographic image with a desired background to createa background replaced image, a background replacing program that causesa computer to operate as the background replacing apparatus, when thebackground replacing program is incorporated into the computer and isexecuted, and a background replacing method capable of naturallyreplacing a background of a photographic image with a desired backgroundto create a background replaced image.

To achieve the above-mentioned object, the present invention provides afirst background replacing apparatus comprising:

an image obtaining section that obtains a plurality of photographicimages generated through photography of a common subject under aplurality of photographic conditions;

an area discriminating section that discriminates among a subject area,a background area, and a boundary area of the photographic image inaccordance with at least one of the photographic images obtained by theimage obtaining section;

a mixing ratio determining section that determines a mixing ratio of asubject color to a background color in the boundary area; and

a background replacing section that replaces a background of thephotographic image obtained by the image obtaining section with adifferent background in accordance with three areas of the subject area,the background area, and the boundary area, which are discriminated bythe area discriminating section, and the mixing ratio determined by themixing ratio determining section.

According to a photographic image offering such a state that abackground appears through a part of a subject, in many cases, an imageportion offering such the state is constructed of pixels each having amixed color of a color of the subject and a color of the backgroundappearing at the back of the subject. Hence, simply replacing thebackground area of the photographic image by an image of a differentbackground would retain the color of the background on the image portionas mentioned above. The reason why an image, in which a background isreplaced in accordance with the chromakey processing and the like, isunnaturally seen for a person is due to this.

According to the first background replacing apparatus of the presentinvention, the area discriminating section discriminates among a subjectarea, a background area, and a boundary area of the photographic imagein accordance with at least one of the photographic images obtained bythe image obtaining section, the mixing ratio determining sectiondetermines a mixing ratio of a subject color to a background color inthe boundary area, and the background replacing section replaces abackground of the photographic image obtained by the image obtainingsection with a different background in accordance with three areas ofthe subject area, the background area, and the boundary area, which arediscriminated by the area discriminating section, and the mixing ratiodetermined by the mixing ratio determining section. Thus, there iscreated an image that causes a person to recognize the state that abackground after replacement appears through a part of a subject. Ineffect, according to the background replacing apparatus of the presentinvention, it is possible to create a background replaced image in whichthe background is naturally replaced by a background of a client wish.

In the first background replacing apparatus according to the presentinvention as mentioned above, it is preferable that the areadiscriminating section discriminates among the subject area, thebackground area, and the boundary area of the photographic image inaccordance with a predetermined different point between the subject andthe background in the photographic conditions.

For example, in the photographic condition that the subject isphotographed in the backlight, there exists an apparent distinctionaddressed as luminous intensity between the subject and the background.According to the background replacing apparatus of the presentinvention, it is possible to readily distinguish the subject area, thebackground area, and the boundary area of the photographic image inaccordance with such a distinction.

In the first background replacing apparatus according to the presentinvention as mentioned above, it is preferable that the areadiscriminating section discriminates the photographic image between asubject-side area and a background-side area in accordance with apredetermined different point between the subject and the background inthe photographic conditions, and at least one of the subject-side areaand the background-side area is reduced so as to be treated as thesubject area and the background area, and an area between the subjectarea and the background area is treated as the boundary area.

In the first background replacing apparatus according to the presentinvention as mentioned above, it is preferable that the areadiscriminating section discriminates the photographic image between asubject-side area and a background-side area in accordance with apredetermined different point between the subject and the background inthe photographic conditions, and the subject-side area is reduced so asto be treated as the subject area, and an area portion excepting thesubject area, of the subject-side area before reduction, is treated aswhole or part of the boundary area.

In the first background replacing apparatus according to the presentinvention as mentioned above, it is preferable that the areadiscriminating section discriminates the photographic image between asubject-side area and a background-side area in accordance with apredetermined different point between the subject and the background inthe photographic conditions, and the background-side area is reduced soas to be treated as the background area, and an area portion exceptingthe background area, of the background-side area before reduction, istreated as whole or part of the boundary area.

According to the background replacing apparatus of the present inventionas mentioned above, after the photographic image is once discriminatedbetween a subject-side area and a background-side area in accordancewith a predetermined different point between the subject and thebackground in the photographic conditions, both or one of those areas isreduced, so that there is set up a suitable boundary area in such amanner that a portion offering a state that the background appearsthrough a part of the subject, which may be seen at the vicinity of theboundary between the subject and the background in the photographicimage, is surely included in the boundary area. This feature makes itpossible to surely suppress the above-mentioned unnaturalness.

In the first background replacing apparatus according to the presentinvention as mentioned above, it is acceptable that the areadiscriminating section discriminates among the subject area, thebackground area, and the boundary area of the photographic image inaccordance with one of the photographic images obtained by the imageobtaining section, and

the background replacing section replaces a background of a photographicimage, which is different from the photographic image used fordiscrimination of areas by the area discriminating section, of thephotographic images obtained by the image obtaining section.

As the typical example of the background replacing apparatus asmentioned above, there is raised an apparatus in which there areobtained a first photographic image having a great different pointbetween a subject and a background in the single photographic image, forexample, a back light photographic image that is obtained through a backlight photography, and a second photographic image that is obtainedthrough a photography of the subject common to the first photographicimage under photographic conditions that a clear photograph of thesubject can be obtained, and first, the subject, the subject area, thebackground area, and the boundary area in the first photographic imageare distinguished from one another in accordance with said greatdifferent point, and then the distinction is applied to the secondphotographic image.

In the first background replacing apparatus according to the presentinvention as mentioned above, it is preferable that the mixing ratiodetermining section presumes the subject color and the mixing ratio inthe boundary area of the photographic image; and

the background replacing section replaces the background of thephotographic image with the different background using the subject colorand the mixing ratio, which are presumed by the mixing ratio determiningsection.

The mixing ratio of the subject color to the background color in theboundary area of the photographic image corresponds to a degree of anappearance of the background in a state that the background appearsthrough a part of the subject. According to the background replacingapparatus of the present invention as mentioned above, there is presumedthe subject color as well as mixing ratio, and thus it is possible todetermine a suitable mixed color and thereby creating a naturalbackground replaced image.

In the first background replacing apparatus according to the presentinvention as mentioned above, it is preferable that the mixing ratiodetermining section presumes the subject color using colors in thesubject area as candidates for the subject color.

According to the background replacing apparatus of the preferable formof the invention as mentioned above, as the candidates for the subjectcolor, there is used the respective color of the subject area, which isdeemed to include a color close to the subject color with highprobability, and thus it is possible to presume the subject color withgreat accuracy.

In the first background replacing apparatus according to the presentinvention as mentioned above, it is preferable that the mixing ratiodetermining section presumes the subject color and the mixing ratio onindividual points in the boundary area, and presumes the subject coloron assumption that the subject color lines up on a straight line basiswith the background color and a color of the associated point in apredetermined color space.

It is considered that a mixed color of two colors is lined up on astraight-line basis in the two colors that are the base of the mixedcolor in a color space, for example, R (red) G (green) B (blue) colorspace. According to the background replacing apparatus of the preferableform of the invention as mentioned above, the subject color is presumedtaking the background color, and the color of the associated point thatis the mixed color of the background color and the subject color, asbeing known, and taking the color lined up with those two colors on astraight-line basis as candidates, and thus it is possible to presumethe subject color with great accuracy.

In the first background replacing apparatus according to the presentinvention as mentioned above, it is preferable that the mixing ratiodetermining section presumes the subject color and the mixing ratio onindividual points in the boundary area, and presumes the subject color,using colors in the subject area as candidates for the subject color, insuch a manner that of the candidates, a candidate, which lines up on astraight line basis with the background color and a color of theassociated point in a predetermined color space, and is closest to thecolor of the associated point, is presumed as the subject color.

Colors of the individual points are colors in which the subject color ismixed with the background color, and thus it is expected that the colorsof the individual points and the subject color have similarity in someextent. According to the first background replacing apparatus of thepresent invention as mentioned above, the candidate closest to thecolors of the individual points is presumed as the subject color, andthus it is possible to presume the subject color with great accuracy.

In the first background replacing apparatus according to the presentinvention as mentioned above, it is preferable that the mixing ratiodetermining section presumes the subject color and the mixing ratio onindividual points in the boundary area, and presumes the mixing ratio inaccordance with a ratio of a mutual distance among the background color,a color of the associated point, and the subject color, in apredetermined color space.

According to the background replacing apparatus of the present inventionas mentioned above, it is possible to readily presume the mixing ratioin accordance with a ratio of a mutual distance among the backgroundcolor, the color of the associated point, and the subject color, in apredetermined color space, which is obtained through a simplecomputation.

In the first background replacing apparatus according to the presentinvention as mentioned above, it is preferable that the backgroundreplacing section replaces the background of the photographic image withthe different background in such a manner that a color of the differentbackground and the subject color presumed by the mixing ratiodetermining section are mixed at the mixing ratio presumed by the mixingratio determining section.

According to a photographic image offering such a state that abackground appears through a part of a subject, in many cases, a colorof a portion offering such the state is a mixed color of the color ofthe background and the subject color. In this case, the mixing ratio ofthe mixed color reflects a state of appearance of the backgroundappearing through a part of the subject. According to the firstbackground replacing apparatus according to the present invention asmentioned above, a color of the different background and the subjectcolor presumed by the mixing ratio determining section are mixed at themixing ratio presumed by the mixing ratio determining section. Thisfeature makes it possible to readily reproduce the state of appearanceof the background appearing through a part of the subject, with theimage after the replacement for the background.

In the first background replacing apparatus according to the presentinvention as mentioned above, it is preferable that the backgroundreplacing apparatus further comprises an image correcting sectionperforms correction of the area discriminated by the area discriminatingsection in response to an operation, and causes the background replacingsection to perform the background replacing based on a corrected area sothat a background replaced image is corrected.

According to the background replacing apparatus of the preferred form ofthe present invention as mentioned above, even if the backgroundreplaced image once created is, for example, an unnatural image in whicha part of green of the subject is replaced together with the backgroundby a different background, it is possible to correct the unnaturalbackground replaced image into a natural background replaced image insuch a manner that the correction of the area discriminated by the areadiscriminating section is carried out, and for example, the imageportion, which includes elements of both the background and the subject,is removed from the background area so that processing for the especialimage portion is performed.

In the background replacing apparatus having the image correctingsection according to the present invention as mentioned above, it isacceptable that the area discriminating section discriminates thephotographic image between a subject-side area and a background-sidearea in accordance with a predetermined different point between thesubject and the background in the photographic conditions, and thesubject-side area and the background-side area are reduced by apredetermined reduction amount so as to be treated as the subject areaand the background area, and

the image correcting section alters the reduction amount on the areadiscriminating section in response to an operation, so that the subjectarea and/or the background area are corrected.

In the background replacing apparatus having the image correctingsection according to the present invention as mentioned above, it isacceptable that the area discriminating section discriminates betweenthe subject area and the background area in accordance with adiscrimination basis based on a predetermined different point betweenthe subject and the background in the photographic conditions, and

the image correcting section alters the discrimination basis in responseto an operation, so that the subject area and/or the background area arecorrected.

According to the background replacing apparatus of the preferred form ofthe present invention as mentioned above, the subject area and/or thebackground area are not directly corrected, but parameters, such as thereduction amount and the discrimination basis, which are sensuouslyrealizable for a person, are altered, and thus it is possible to easilycorrect the subject area and/or the background area.

To achieve the above-mentioned object, the present invention provides asecond background replacing apparatus comprising:

an image obtaining section that obtains a plurality of photographicimages generated through photography of a common subject under aplurality of photographic conditions;

an area discriminating section that discriminates a background area ofthe photographic image from other areas in accordance with at least oneof the photographic images obtained by the image obtaining section;

a mixing state presuming section that presumes a mixing ratio of asubject color and a background color in other areas excepting thebackground area in the photographic image, and the subject color; and

a background replacing section that replaces a background of thephotographic image obtained by the image obtaining section with adifferent background using the subject color and the mixing ratio, whichare presumed by the mixing state presuming section.

According to the second background replacing apparatus of the presentinvention, the area discriminating section discriminates a backgroundarea of the photographic image from other areas in accordance with atleast one of the photographic images obtained by the image obtainingsection, and the mixing state presuming section presumes a mixing ratioof a subject color and a background color in other areas excepting thebackground area in the photographic image, and the subject color. Themixing ratio corresponds to a degree of an appearance of the backgroundin a state that the background appears through a part of the subject.According to the background replacing apparatus of the present inventionas mentioned above, there is presumed the subject color as well asmixing ratio, and thus it is possible to determine a suitable mixedcolor and thereby creating a natural background replaced image. Ineffect, according to the background replacing apparatus of the presentinvention, it is possible to create a background replaced image in whichthe background is naturally replaced by a background of a client wish.

C17 In the second background replacing apparatus according to thepresent invention as mentioned above, it is preferable that the mixingstate presuming section presumes the subject color using colors in thesubject area as candidates for the subject color.

According to the background replacing apparatus of the present inventionas mentioned above, as the candidates for the subject color, there isused the respective color of the photographic image, which is deemed toinclude a color close to the subject color with high probability, andthus it is possible to presume the subject color with great accuracy.

In the second background replacing apparatus according to the presentinvention as mentioned above, it is preferable that the mixing statepresuming section presumes the subject color and the mixing ratio onindividual points in other areas excepting the background area, andpresumes the subject color on assumption that the subject color lines upon a straight line basis with the background color and a color of theassociated point in a predetermined color space.

It is considered that a mixed color of two colors is lined up on astraight-line basis in the two colors that are the base of the mixedcolor in a color space, for example, R (red) G (green) B (blue) colorspace. According to the second background replacing apparatus of thepresent invention as mentioned above, the subject color is presumedtaking the background color and the color of the individual points,which is the mixed color of the background color and the subject color,as being known, and taking the color lined up with those two colors on astraight-line basis as candidates, and thus it is possible to presumethe subject color with great accuracy.

In the second background replacing apparatus according to the presentinvention as mentioned above, it is preferable that the mixing statepresuming section presumes the subject color and the mixing ratio onindividual points in other areas excepting the background area, andpresumes the subject color, using colors in other areas excepting thebackground area as candidates for the subject color, in such a mannerthat of the candidates, a candidate, which lines up on a straight linebasis with the background color and a color of the associated point in apredetermined color space, and is farthest from the color of theassociated point, is presumed as the subject color.

It is considered that the other areas excepting the background areaincludes a color close to the subject color with high probability, andalso includes additional color mixed at other mixing ratio between thesubject color and a color of the individual points with highprobability. According to the second background replacing apparatus ofthe present invention as mentioned above, the mixing state presumingsection presumes a candidate, which is farthest from the color of theassociated point, as the subject color in other areas excepting thebackground area. Thus, it is possible to avoid such an additionalconfusing color and the like, and thereby presuming the subject colorwith great accuracy.

In the second background replacing apparatus according to the presentinvention as mentioned above, it is preferable that the mixing statepresuming section presumes the subject color and the mixing ratio onindividual points in other areas excepting the background area, andpresumes the mixing ratio in accordance with a ratio of a mutualdistance among the background color, a color of the associated point,and the subject color, in a predetermined color space.

According to the background replacing apparatus of the present inventionas mentioned above, it is possible to readily presume the mixing ratioin accordance with a ratio of a mutual distance among the backgroundcolor, a color of the associated point, and the subject color, in apredetermined color space, which is obtained through a simplecomputation.

In the second background replacing apparatus according to the presentinvention as mentioned above, it is preferable that the backgroundreplacing section replaces the background of the photographic image withthe different background in such a manner that a color of the differentbackground and the subject color presumed by the mixing state presumingsection are mixed at the mixing ratio presumed by the mixing statepresuming section.

According to a photographic image offering such a state that abackground appears through a part of a subject, in many cases, a colorof a portion offering such the state is a mixed color of the color ofthe background and the subject color. In this case, the mixing ratio ofthe mixed color reflects a state of appearance of the backgroundappearing through a part of the subject. According to the backgroundreplacing apparatus of the preferred form of the present invention asmentioned above, a color of the different background and the subjectcolor presumed by the mixing ratio determining section are mixed at themixing ratio presumed by the mixing ratio determining section. Thisfeature makes it possible to readily reproduce the state of appearanceof the background appearing through a part of the subject, with theimage after the replacement for the background.

In the background replacing apparatus according to the present inventionas mentioned above, it is preferable that the area discriminatingsection discriminates a background area of the photographic image fromother areas in accordance with a ratio of luminous intensity among theplural photographic images.

As one way of distinguishing a background area of a photographic imagefrom other area, there is considered, for example, a method ofdistinguishing the background area of the photographic image from otherarea in accordance with a difference in luminous intensity between thebackground and the subject in the photographic image. However, forexample, in the event that the photographic image is photographed underthe light illumination, there is no substantial difference in luminousintensity between the background and the subject. Thus, according tosuch a method, it happens that it is difficult to distinguish thebackground area from other area. According to the background replacingapparatus of the preferable form of the invention as mentioned above,distinguishing the background area of the photographic image from otherarea is performed in accordance with a ratio of luminous intensity amongthe plural photographic images. Thus, according to this distinguishingmethod, a preparation of a plurality of photographic images mutuallydifferent in luminous intensity of the background for instance makes itpossible to preferably distinguish the background area of thephotographic image from other area, since the background area isdifferent from other area in a ratio of luminous intensity, even ifthere is no substantial difference in luminous intensity between thebackground and the subject in a sheet of photographic image.

In the background replacing apparatus according to the present inventionas mentioned above, it is preferable that the background replacingapparatus further comprises a storage section that stores the subjectcolor and the mixing ratio, which are presumed by the mixing statepresuming section, and the a background replacing section replaces abackground of the photographic image obtained by the image obtainingsection with a different background using the subject color and themixing ratio, which are stored in the storage section.

According to the background replacing apparatus of the preferred form ofthe present invention as mentioned above, for example, in the event thatafter a background is replaced once, the replaced background is replacedwith another background, it is possible to save one trouble forcomputation of the subject color and the mixing ratio.

In the background replacing apparatus according to the present inventionas mentioned above, as a first form, it is preferable that thebackground replacing apparatus further comprises a first parameterdetermining section that determines a correction parameter to be usedfor a color correction for the subject color in accordance with an imageof said other area in which a color is replaced by the subject colorpresumed in the mixing state presuming section, and

a first color correcting section that applies the color correction tothe subject color, which is presumed by the mixing state presumingsection, using the correction parameter determined by the firstparameter determining section,

wherein the background replacing section replaces the background of thephotographic image obtained by the image obtaining section with adifferent background using the subject color subjected to the colorcorrection by the first color correcting section and the mixing ratio.

In the background replacing apparatus according to the present inventionas mentioned above, as a second form, it is preferable that thebackground replacing apparatus further comprises

a second parameter determining section that determines a correctionparameter to be used for a color correction for the subject color inaccordance with the photographic image obtained by the image obtainingsection, and

a second color correcting section that applies the color correction tothe subject color, which is presumed by the mixing state presumingsection, using the correction parameter determined by the secondparameter determining section,

wherein the background replacing section replaces the background of thephotographic image obtained by the image obtaining section with adifferent background using the subject color subjected to the colorcorrection by the second color correcting section and the mixing ratio.

In the background replacing apparatus according to the present inventionas mentioned above, as a third form, it is preferable that thebackground replacing apparatus further comprises

a third parameter determining section that determines a correctionparameter to be used for a color correction for the subject color to aparameter according to an operation, and

a third color correcting section that applies the color correction tothe subject color, which is presumed by the mixing state presumingsection, using the correction parameter determined by the thirdparameter determining section,

wherein the background replacing section replaces the background of thephotographic image obtained by the image obtaining section with adifferent background using the subject color subjected to the colorcorrection by the third color correcting section and the mixing ratio.

There is known a color correction addressed as a so-called set up inwhich the color shade and the white balance of a photographic image arecorrected to the color shade and the white balance which are generallydeemed to be preferable for a person. Apparently, in the replacement ofa background on the photographic image which is subjected to the set up,it seems that a high picture quality of image, in which the backgroundis replaced, can be obtained. However, the use of the photographic imagesubjected to the set up causes the mixing state presuming section toincorrectly perform the presumption of the subject color for the reasonthat for example, the gradation of the image is altered. Thus, there isa possibility that it is difficult to perform the replacement of thenatural background. In the event that the set up is applied to the imageafter the background replacement, the color of the background for thereplacement is inadvertently changed too. According to the backgroundreplacing apparatus of three types of preferred forms of the presentinvention as mentioned above, in any cases, the color correction isapplied to the subject color which is presumed by the mixing statepresuming section. Therefore, according to the background replacingapparatus of those three types of preferred forms of the presentinvention as mentioned above, the adoption of, for example, the set upas the color correction, makes it possible to execute the set up withouthaving an effect on the color of the background for the replacement andthe presumption of the subject color by the mixing state presumingsection, and thereby obtaining a high picture quality of image in whicha background is replaced. It is acceptable that the correction parameterto be used for the set up is determined in accordance with the image onthe other area in which the color is replaced by the subject color, asthe first form of the three types of preferred forms of the presentinvention as mentioned above, or it is determined in accordance with thephotographic image, as the second form, or alternatively, it isdetermined in accordance with the user's operation, as the third form.

In the background replacing apparatus according to the present inventionas mentioned above, it is preferable that the background replacingapparatus further comprises a fourth parameter determining section thatdetermines a correction parameter to be used for a color correction forthe subject color to a parameter according to the different background,and

a fourth color correcting section that applies the color correction tothe subject color, which is presumed by the mixing state presumingsection, using the correction parameter determined by the fourthparameter determining section,

wherein the background replacing section replaces the background of thephotographic image obtained by the image obtaining section with adifferent background using the subject color subjected to the colorcorrection by the fourth color correcting section and the mixing ratio.

According to the background replacing apparatus of the preferred form ofthe present invention as mentioned above, it is possible to perform theset up without having an effect on the presumption of the subject colorand the color of the background for the replacement, and therebyobtaining a high picture quality of image in which a background isreplaced. And in addition, for example, in the event that an eveningscene is used as the different background, when there is performed theset up that the subject color is corrected to a color inclining to anorange color in accordance with the evening scene, it is possible toobtain an image in which the background is more naturally replaced.

In the background replacing apparatus according to the present inventionas mentioned above, it is preferable that the background replacingsection replaces a background of the photographic image using, as thedifferent background, a background to which relation informationrepresentative of a relative positional relation in an image between thesubject and the background is applied, and performs an replacement insuch a manner that the relative relation in the background replacedimage between the subject and the background is the same relativerelation as the relative relation represented by the relationinformation.

According to the background replacing apparatus of the preferred form ofthe present invention as mentioned above, it is possible to perform areplacement of the background so that for example, position and size ofthe subject in the background replaced image are natural ones to thedifferent background, and thereby obtaining an image in which thebackground is more naturally replaced.

To achieve the above-mentioned object, the present invention provides athird background replacing apparatus comprising:

an image obtaining section that obtains a plurality of photographicimages generated through photography of a common subject under aplurality of photographic conditions mutually different in color of thebackground;

an area discriminating section that discriminates a background area ofthe photographic image from other areas in accordance with thephotographic images obtained by the image obtaining section;

a mixing state presuming section that presumes a mixing ratio of thesubject color to the background color, and the subject color, in otherareas excepting the background area in the photographic image, inaccordance with the plurality of photographic images; and

a background replacing section that replaces a background of thephotographic image obtained by the image obtaining section with adifferent background using the subject color and the mixing ratio, whichare presumed by the mixing state presuming section.

According to the third background replacing apparatus of the presentinvention, the mixing state presuming section presumes a mixing ratio ofthe subject color to the background color, and the subject color, inother areas excepting the background area in the photographic image, inaccordance with the plurality of photographic images. The mixing ratiocorresponds to a degree of an appearance of the background in a statethat the background appears through a part of the subject. According tothe background replacing apparatus of the present invention as mentionedabove, there is presumed the subject color as well as mixing ratio, andthus it is possible to determine a suitable mixed color and therebycreating a natural background replaced image. According to thebackground replacing apparatus of the present invention as mentionedabove, the subject color and the mixing ratio are presumed in accordancewith the plurality of photographic images, which are mutually differentin the background color. Since those pluralities of photographic imagesare obtained through photography of the common subject, the subjectcolor is common among those pluralities of photographic images. Thus,according to the background replacing apparatus of the presentinvention, it is possible to perform a presumption of the subject colorand the mixing ratio using the unique and exact presuming scheme, inwhich for example, first, a single subject color is presumed inaccordance with a plurality of presuming materials, such as a pluralityof photographic images, and then the mixing ratio is presumed from thepresumed subject color. In effect, according to the background replacingapparatus of the present invention, it is possible to create abackground replaced image in which the background is naturally replacedby a background of a client wish.

In the third background replacing apparatus according to the presentinvention as mentioned above, it is preferable that the mixing statepresuming section presumes, on assumption that the plurality ofphotographic images are equal to each other in terms of mixing state ofthe subject color and the background color, the subject color and themixing ratio, the subject color and the mixing ratio.

According to the background replacing apparatus of the preferred form ofthe present invention as mentioned above, it is possible to considerthat the mixing ratio in the mixed color is common among the pluralityof photographic images. Thus, also regarding the mixing ratio, it ispossible to perform a presumption of a single mixing ratio in accordancewith the unique and exact presuming scheme, in which a single mixingratio is presumed in accordance with a plurality of presuming materials,such as a plurality of photographic images.

In the third background replacing apparatus according to the presentinvention as mentioned above, it is preferable that the mixing statepresuming section presumes the subject color and the mixing ratio onindividual points in other area excepting the background area,determines straight lines each coupling in a predetermined color spacethe background color with a color of the associated point in each of theplurality of photographic images generated through photography of thecommon subject under a plurality of photographic conditions mutuallydifferent in color of the background, and presumes a color correspondingto an intersection of the straight lines as the subject color.

According to the third background replacing apparatus of the presentinvention as mentioned above, it is possible to mathematically uniquelydetermine a subject color, and then readily determine the mixing ratiofrom the subject color.

In the third background replacing apparatus according to the presentinvention as mentioned above, it is preferable that the image obtainingsection obtains a plurality of photographic images generated throughphotography of the common subject under a plurality of photographicconditions mutually different in color-phase of the background.

According to the background replacing apparatus of the preferred form ofthe present invention as mentioned above, it is possible to presume thesubject color and the mixing ratio with great accuracy.

In the third background replacing apparatus according to the presentinvention as mentioned above, it is preferable that the areadiscriminating section discriminates between a subject area and aboundary area of other areas excepting the background area in thephotographic image; and

the mixing state presuming section that presumes the subject color andthe mixing ratio in the boundary area.

According to the background replacing apparatus of the preferred form ofthe present invention as mentioned above, the limit of the object forthe presuming processing is restricted in the photographic image. Thisfeature makes it possible to efficiently perform the presumptionprocessing.

In the third background replacing apparatus according to the presentinvention as mentioned above, it is preferable that the areadiscriminating section discriminates the background area of thephotographic image obtained by the image obtaining section from otherareas in accordance with an amount of the change in color among theplurality of photographic images.

In the third background replacing apparatus according to the presentinvention as mentioned above, it is preferable that the areadiscriminating section discriminates the background area of thephotographic image obtained by the image obtaining section from otherareas in accordance with a difference of luminous intensity between thebackground and the subject.

Those features as mentioned above make it possible to readily implementthe area discriminating section. According to the former feature inwhich the distinction of the areas is performed in accordance with anamount of the change in color, it is possible to perform the distinctionof the areas using the photographic image as one used in the presumptionfor the subject color and the mixing ratio, and thereby saving thenumber of photographic images needed for the background replacingprocessing in its entirety. According to the latter feature in which thedistinction of the areas is performed in accordance with the differenceof luminous intensity between the background and the subject, there areneeded the photographic image that obtained through photography of thesubject under the photographic condition that luminous intensity of thebackground is different from the luminous intensity of the subject. Onthe other hand, it is possible to perform the distinction of the areasin accordance with only the photographic image, and thereby contributingto simplification of the distinction of the areas itself.

To achieve the above-mentioned object, the present invention provides afirst background replacing program that causes a computer to operate asan apparatus, when the background replacing program is incorporated intothe computer and is executed, the apparatus comprising:

an image obtaining section that obtains a plurality of photographicimages generated through photography of a common subject under aplurality of photographic conditions;

an area discriminating section that discriminates among a subject area,a background area, and a boundary area of the photographic image inaccordance with the photographic images obtained by the image obtainingsection;

a mixing ratio determining section that determines a mixing ratio of asubject color to a background color in the boundary area; and

a background replacing section that replaces a background of thephotographic image obtained by the image obtaining section with adifferent background in accordance with three areas of the subject area,the background area, and the boundary area, which are discriminated bythe area discriminating section, and the mixing ratio determined by themixing ratio determining section.

To achieve the above-mentioned object, the present invention provides asecond background replacing program that causes a computer to operate asan apparatus, when the background replacing program is incorporated intothe computer and is executed, the apparatus comprising:

an image obtaining section that obtains a plurality of photographicimages generated through photography of a common subject under aplurality of photographic conditions;

an area discriminating section that discriminates a background area ofthe photographic image from other areas in accordance with thephotographic images obtained by the image obtaining section;

a mixing state presuming section that presumes a mixing ratio of thesubject color and the background color, and the subject color, in otherareas excepting the background area in the photographic image; and

a background replacing section that replaces a background of thephotographic image obtained by the image obtaining section with adifferent background using the subject color and the mixing ratio, whichare presumed by the mixing state presuming section.

To achieve the above-mentioned object, the present invention provides athird background replacing program that causes a computer to operate asan apparatus, when the background replacing program is incorporated intothe computer and is executed, the apparatus comprising:

an image obtaining section that obtains a plurality of photographicimages generated through photography of a common subject under aplurality of photographic conditions in which colors of backgrounds aredifferent from one another;

an area discriminating section that discriminates a background area ofthe photographic image from other areas in accordance with thephotographic images obtained by the image obtaining section;

a mixing state presuming section that presumes a mixing ratio of thesubject color and the background color, and the subject color, in otherareas excepting the background area in the photographic image, inaccordance with the plurality of photographic images; and

a background replacing section that replaces a background of thephotographic image obtained by the image obtaining section with adifferent background using the subject color and the mixing ratio, whichare presumed by the mixing state presuming section.

According to the first to third background replacing programs of thepresent invention as mentioned above, it is possible to readilyimplement a background replacing apparatus capable of creating abackground replaced image in which the background of the photographicimage is naturally replaced with the desired background.

To achieve the above-mentioned object, the present invention provides afirst background replacing method comprising:

an image obtaining step that obtains a plurality of photographic imagesgenerated through photography of a common subject under a plurality ofphotographic conditions;

an area discriminating step that discriminates among a subject area, abackground area, and a boundary area of the photographic image inaccordance with the photographic images obtained by the image obtainingstep;

a mixing ratio determining step that determines a mixing ratio of asubject color to a background color in the boundary area; and

a background replacing step that replaces a background of thephotographic image obtained by the image obtaining step with a differentbackground in accordance with three areas of the subject area, thebackground area, and the boundary area, which are discriminated by thearea discriminating step, and the mixing ratio determined by the mixingratio determining step.

To achieve the above-mentioned object, the present invention provides asecond background replacing method comprising:

an image obtaining step that obtains a plurality of photographic imagesgenerated through photography of a common subject under a plurality ofphotographic conditions;

an area discriminating step that discriminates a background area of thephotographic image from other areas in accordance with the photographicimages obtained by the image obtaining step;

a mixing state presuming step that presumes a mixing ratio of a subjectcolor and a background color, and the subject color, in other areasexcepting the background area in the photographic image; and

a background replacing step that replaces a background of thephotographic image obtained by the image obtaining step with a differentbackground using the subject color and the mixing ratio, which arepresumed by the mixing state presuming step.

To achieve the above-mentioned object, the present invention provides athird background replacing method comprising:

an image obtaining step that obtains a plurality of photographic imagesgenerated through photography of a common subject under a plurality ofphotographic conditions;

an area discriminating step that distinguishes a background area of thephotographic image from other areas in accordance with the photographicimages obtained by the image obtaining step;

a mixing state presuming step that presumes a mixing ratio of a subjectcolor and a background color, and the subject color, in other areasexcepting the background area in the photographic image, in accordancewith the plurality of photographic images; and

a background replacing step that replaces a background of thephotographic image obtained by the image obtaining step with a differentbackground using the subject color and the mixing ratio, which arepresumed by the mixing state presuming step.

According to the first to third background replacing methods of thepresent invention as mentioned above, it is possible to readily createthe background replaced image in which the background is naturallyreplaced with the desired background.

The background replacing program storage medium includes all the aspectscorresponding to those of the background replacing apparatus.

As mentioned above, according to the present invention, it is possibleto create a background replaced image in which a background of aphotographic image is naturally replaced with a desired background.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a construction view of an image taking system, to which anembodiment of a background replacing apparatus of the present inventionis applied.

FIG. 2 is a flowchart useful for understanding a flow of work in whichan embodiment of a background replacing method of the present invention,which is carried out by the image taking system shown in FIG. 1, isapplied.

FIG. 3 is a hardware structural view of the personal computer shown inFIG. 1.

FIG. 4 is a view useful for understanding an embodiment of a backgroundreplacing program related to the present invention.

FIG. 5 is a functional block diagram of elements for operating thepersonal computer shown in FIG. 1 and FIG. 3 in form of an embodiment ofa background replacing apparatus of the present invention, when thebackground replacing program shown in FIG. 4 is installed in thepersonal computer shown in FIG. 1 and FIG. 3.

FIG. 6 is an explanatory view useful for understanding details of theimage correcting section shown in FIG. 5.

FIG. 7 is a typical illustration showing an example of a follow lightphotographic image involving a trapezoidal distortion.

FIG. 8 is a typical illustration showing a state that the trapezoidaldistortion of the follow light photographic image of FIG. 7 iscorrected.

FIG. 9 is a typical illustration showing an effective image that iscreated in accordance with the follow light photographic image correctedin the trapezoidal distortion, shown in FIG. 8.

FIG. 10 is a typical illustration showing a corrected back lightphotographic image, which is subjected to image correction processing bythe image correcting section of FIG. 6.

FIG. 11 is an illustration showing an example of the corrected followlight photographic image.

FIG. 12 is an illustration showing an example of the corrected backlight photographic image.

FIG. 13 is an enlarged view of the area A1 shown in FIG. 11.

FIG. 14 is an enlarged view of the area A1 shown in FIG. 12.

FIG. 15 is an explanatory view useful for understanding details of thereplacement processing section shown in FIG. 5.

FIG. 16 is an explanatory view useful for understanding details of themask creating section shown in FIG. 15.

FIG. 17 is a view showing various examples of histograms as to luminousintensity of pixels constituting the corrected back light photographicimage.

FIG. 18 is a view showing an example of a histogram different from theexample of the histogram shown in FIG. 17.

FIG. 19 is an illustration showing an area that is set up as an initialsubject mask and an area that is set up as an initial background mask,in the enlarged view of the back light photographic image shown in FIG.14.

FIG. 20 is an illustration showing the initial subject mask and theinitial background mask shown in FIG. 19.

FIG. 21 is an illustration showing a state that the initial subject maskand the initial background mask are overlapped on the corrected followlight photographic image shown in FIG. 11 in its entirety.

FIG. 22 is an illustration showing a state that the initial subject maskand the initial background mask shown in FIG. 20 are reduced in adirection going away from the boundary area, respectively.

FIG. 23 is an illustration showing a state that the subject mask and thebackground mask, which are finally completed, are overlapped on thecorrected follow light photographic image shown in FIG. 11 in itsentirety.

FIG. 24 is a view showing a mask creating section, which is differentfrom the mask creating section 731 of the present embodiment.

FIG. 25 is a view showing a state that the corrected back lightphotographic image is divided.

FIG. 26 is a view showing examples of the histograms, which are createdon the three small areas Ar1, Ar2, and Ar3 shown in FIG. 25,respectively.

FIG. 27 is an explanatory view useful for understanding details of thecolor presuming section shown in FIG. 15.

FIG. 28 is an explanatory view useful for understanding details of thebackground color presuming section shown in FIG. 27.

FIG. 29 is an illustration useful for understanding a state that when abackground color is estimated on a certain pixel, a search range is setup on the pixel.

FIG. 30 is an illustration showing an example of an area used forpresuming a background color on a certain pixel.

FIG. 31 is an explanatory view useful for understanding details of thesubject color presuming section shown in FIG. 27.

FIG. 32 is an illustration useful for understanding a state that when asubject color is presumed on a certain pixel, a search range is set upon the pixel.

FIG. 33 is an illustration showing an example of an area used forpresuming a subject color on a certain pixel.

FIG. 34 is an illustration useful for understanding a method ofsearching a subject color on a certain pixel in a boundary area of thecorrected follow light photographic image.

FIG. 35 is an illustration useful for understanding a scheme of acombination color on a certain pixel in a boundary area of the correctedfollow light photographic image.

FIG. 36 is a view showing an example of a background replaced image.

FIG. 37 is a view showing details of the image revision section 760shown in FIG. 5.

FIG. 38 is a view showing an image revision section which is differentfrom the image revision section shown in FIG. 37.

FIG. 39 is a typical illustration useful understanding a state that thepresumption of the subject color and the combination ratio α is carriedout from pixels closer to the area defined by the subject mask M1′ inthe boundary area.

FIG. 40 is a typical illustration useful understanding a state thatsetting up of the retrieval range is carried out using the apparentoutlines.

FIG. 41 is an illustration useful for understanding a state that when asubject color is estimated on a certain pixel in an area defined by anon-background mask, of the corrected follow light photographic image, asearch range is set up on the pixel.

FIG. 42 is an illustration showing an example of an area used forestimating a subject color on a certain pixel.

FIG. 43 is an illustration useful for understanding a method ofsearching a subject color on a certain pixel in an area defined by anon-background mask, of the corrected follow light photographic image.

FIG. 44 is a view of a replacement processing section that is capable ofeffectively coping with the plurality of backgrounds to be replaced,which is different from the replacement processing section 730 shown inFIG. 15.

FIG. 45 is a view of a replacement processing section having a colorcorrecting section for applying the color correcting processing to thepresumed subject color, which is different from the replacementprocessing section 730 shown in FIG. 15.

FIG. 46 is a construction view of an image taking system, to which asecond embodiment of a background replacing apparatus of the presentinvention is applied.

FIG. 47 is a flowchart useful for understanding a second embodiment of abackground replacing method of the present invention, which is carriedout by the image taking system shown in FIG. 46.

FIG. 48 is a view useful for understanding a second embodiment of abackground replacing program storage medium of the present inventionstoring a second embodiment of a background replacing program related tothe present invention.

FIG. 49 is a functional block diagram of elements for operating thepersonal computer shown in FIG. 46 and FIG. 3 in form of a secondembodiment of a background replacing apparatus of the present invention,when the background replacing program shown in FIG. 48 is installed inthe personal computer shown in FIG. 46.

FIG. 50 is an explanatory view useful for understanding details of thereplacement processing section shown in FIG. 49.

FIG. 51 is an explanatory view useful for understanding details of thecolor presuming section shown in FIG. 50.

FIG. 52 is an illustration useful for understanding a method ofestimating a subject color that is the origin of colors of pixels in aboundary area, which is common between the corrected follow lightphotographic images that are mutually different in a background color.

BEST MODE FOR CARRYING OUT THE INVENTION

Embodiments of the present invention will be described with reference tothe accompanying drawings.

FIG. 1 is a construction view of an image taking system, to which anembodiment of a background replacing apparatus of the present inventionis applied.

An image taking system 1 shown in FIG. 1 comprises: a photo-studio 10for obtaining a photographic image through photography of the subject; apersonal computer 20 that is operated in form of a first embodiment of abackground replacing apparatus of the present invention, in which thesubject of the photographic image is distinguished from the background,so that the background of the photographic image is replaced with adesired background to create a background replaced image; a printer 30;and a server 40. The photo-studio 10 comprises a digital camera 11, aflashlight emitting device 12, an EL (Electro-Luminescence) panel 130,and a power source 14 for driving the EL panel.

The digital camera 11 has a multi-page function, in which when anoperator depresses a shutter button once, twice photography for asubject P is automatically carried out for a short time. A photographicimage, which is obtained through a first photography, is temporarilystored in a memory of the digital camera 11. In the image taking system1, the digital camera 11 is connected to the personal computer 20, andat the time point that a second photography is completed, a photographicimage, which is obtained through the second photography, is directlytransmitted to the personal computer 20. The digital camera 11 is alsoconnected to the flashlight emitting device 12, so that the digitalcamera 11 transmits to the flashlight emitting device 12 a flashlightinstruction signal to emit a flashlight for every photography.

According to the present embodiment, it is intended to contribute tosaving space on the photo-studio 10, and thus the digital camera 11 willtake a picture of the subject from a short distance looking down on thesubject. As a result, there is a high possibility that a photographicimage, which is obtained through photography by the digital camera 11,involves a so-called trapezoidal distortion in which a width of an imageis more narrowed as it goes more downward. According to the presentembodiment, the trapezoidal distortion is corrected by the personalcomputer 20.

Upon receipt of the flashlight instruction signal from the digitalcamera 11, the flashlight emitting device 12 emits flashlight to thesubject P. There is a need for the flashlight emitting device 12 to becharged for every light emission, and it takes a time up to thesubsequent light emission after once light emission. While the digitalcamera 11 transmits to the flashlight emitting device 12 the flashlightinstruction signal every photography in twice photography, theflashlight emitting device 12 emits the flashlight at only the time ofthe first photography, and thus the flashlight emitting device 12 turnsoff at the time of the first photography, since it is not enough forcharging. The flashlight, which is emitted from the flashlight emittingdevice 12, is high luminance of light such as about 2000 cd/m² inluminance of emitted light.

The EL panel 130 comprises: a cover member 131 having a loading plane131 a for loading with the subject P and a rear plane 131 b located atthe back of the subject P, which are transparent; and a dispersion typeof EL device 132 that is housed in the cover member 131. A power source14 serves as a power source for applying a driving voltage to thedispersion type of EL device 132. On the four corners of the rear plane131 b of the cover member 131, there are provided four makers 131 c eachindicating the range of the rear plane 131 b. The four makers 131 cappear the photographic image together with the subject P, so as to beutilized for correction of the trapezoidal distortion for thephotographic image.

The dispersion type of EL device 132 is a panel light emission lightsource, which is shaped as a sheet, wherein one in which fluorescentpowder is dispersed in a binder having a high dielectric constant isinterposed between two electrodes each shaped as a sheet in which as asubstrate, a flexible plastic is adopted. The dispersion type of ELdevice 132 emits light when an AC voltage from the power source 14 isapplied between the two electrodes.

The dispersion type of EL device 132 is a light source that is very thinsuch as several hundreds of u meters to 1 millimeter in thickness, andis light. Thus, the dispersion type of EL device 132 can be readilyinstalled in a place having no space in thickness, such as inside of thecover member 131. Further, the dispersion type of EL device 132 hasvarious advantages in a property, such that generation of heat at thetime of light emission is less on the order of 2% C, a response speedfrom a start of light emission up to the maximum luminance is high, thelife of a light emission is substantially constant and thus the planningexchange of the device is possible, a local light emission is possible,it withstands impact and vibrations, and dissipation power is small suchas 50 W/m² (at the time of the application of AC power of 50 Hz).Furthermore, the dispersion type of EL device 132 has an advantage inthe economy such that the manufacturing cost is inexpensive since it ismanufactured through a simple manufacturing process.

According to the general dispersion type of EL device, it is possible toadjust the luminescent color to various colors including white throughfurther mixing of a plurality of fluorescent powders mutually differentfrom one another in luminescent color at the time of the manufacture. Tothe contrary, according to the dispersion type of EL device 132 of thepresent embodiment, the fluorescent powders for the color adjustment arenot mixed. As a result, the dispersion type of EL device 132 has two ormore luminescence peaks during 400 nm to 530 nm in luminescencewavelength. This light is concerned with colors between blue and green,which are the original colors of the dispersion type of EL device. Thedispersion type of EL device 132 of the present embodiment emits lightof blue and green colors consisting of two or more luminescence peaks asmentioned above. The blue and green colors are complementary colors fora color of a person's skin, and therefore, according to the image takingsystem 1 of the present embodiment, when photography of a person istaken as the subject, it is possible to obtain a photographic image inwhich the background is blue and green colors, and thereby making itpossible to distinguish the subject and the background in thephotographic image with great accuracy. There is a tendency that thefluorescent powders for the color adjustment lower luminance ofluminescent color. To the contrary, according to the dispersion type ofEL device 132 of the present embodiment, such fluorescent powders forthe color adjustment are not mixed, and thus the dispersion type of ELdevice 132 may emit light with the original luminance of the dispersiontype of EL device.

The power source 14 can control the AC voltage to be applied to thedispersion type of EL device 132 in ranges of 50 Hz to 10 kHz infrequency and 40V to 300V in magnitude. The luminance of light emissionof the dispersion type of EL device 132 is substantially proportional tothe magnitude of the AC voltage to be applied to the dispersion type ofEL device 132. According to the photo-studio 10 of the presentembodiment, the AC voltage of the power source 14 is adjusted in rangesof 1.2 kHz to 1.5 kHz in frequency and 130V to 300V in magnitude, sothat the luminance of light emission of the dispersion type of EL device132 is adjusted in range of 500 cd/m² to 600 cd/m².

According to the photo-studio 10 of the present embodiment, photographyby the digital camera 11 is carried out in a condition that the EL panel130 always turns on. As mentioned above, the first photography iscarried out in the state that the flashlight emitting device 12 emitsflashlight, and the second photography is carried out in the state thatthe flashlight emitting device 12 turns off. Further, according to thephoto-studio 10 of the present embodiment, the luminance of theflashlight emitted from the flashlight emitting device 12 is extremelyhigher as compared with luminance when the EL panel 130 turns on. As aresult, according to the first photography, there is obtained a followlight photographic image, in which the subject P is clearlyphotographed, through photography in a follow light state that aflashlight, which is higher in luminance than the luminance of the ELpanel 130, is projected from the side of the digital camera 11 to thesubject P. According to the second photography to be carried outsubsequent to the first photography, there is obtained a back lightphotographic image, in which an image of the subject P is biased to theshadow side and an image of the EL panel 130 is biased to the highlightside, through photography in a back light state that only light from theEL panel 130 is projected from the back to the subject P.

The personal computer 20 serves as the first embodiment of thebackground replacing apparatus of the present invention. The personalcomputer 20 applies image correction processing including the correctionof the trapezoidal distortion to the two photographic images transferredfrom the digital camera 11, and in addition performs backgroundreplacing processing for creating a background replaced image inaccordance with the corrected photographic images. The image correctionprocessing and the background replacing processing will be describedlater.

The personal computer 20 comprises, on an external appearance, a mainframe unit 210 having a flexible disk-mounting slot for mounting aflexible disk (FD) and a CD-ROM mounting slot for mounting a CD-ROM, themain frame unit 210 having access functions of accessing to the mountedFD and CD-ROM, an image display unit 220 for displaying an image on adisplay screen 220 a in accordance with an instruction from the mainframe unit 210, a keyboard 230 for inputting various sorts ofinformation such as user's instruction and character information to themain frame unit 210 in accordance with a key operation, a mouse 240 forinputting an instruction according to, for example, an icon and thelike, through designation of an optional position on the display screen220 a, the icon and the like being displayed on the position on thedisplay screen 220 a, and a media drive 250 on which a small type ofstorage media, which is used for storing photographic images in digitalcamera and the like, is mounted, the media drive 250 accessing the smalltype of storage media.

The printer 30 prints images transmitted from the personal computer 20.According to the image taking system 1, the printer 30 prints thebackground replaced image created through the background replacingprocessing by the personal computer 20.

The server 40 stores a plurality of sorts of backgrounds, which is usedin the background replacing processing to be executed by the personalcomputer 20. The backgrounds stored in the server 40 may be displayed onthe display screen 220 a. According to the image taking system 1, it ispossible to display on the display screen 220 a backgrounds that areprovided via input storage media such as the CD-ROM and the small typeof storage media, as well as the backgrounds stored in the server 40. Acustomer can select a desired background from among the backgroundsdisplayed on the display screen 220 a.

Next, there will be explained a flow of work in which an embodiment of abackground replacing method of the present invention, which is carriedout by the image taking system shown in FIG. 1, is applied. In thefollowing figures, the same reference numbers as those of FIG. 1 denotesthe same parts.

FIG. 2 is a flowchart useful for understanding a flow of work in whichan embodiment of a background replacing method of the present invention,which is carried out by the image taking system 1 shown in FIG. 1, isapplied.

The work shown in the flowchart of FIG. 2 is carried out on theassumption that in the photo-studio 10, the subject P, the digitalcamera 11 and the flashlight emitting device 12 are arranged at theirsuitable positions, respectively.

First, a voltage is applied from the power source 14 to the dispersiontype of EL device 132 to turn on the EL panel 130 (step S101). Next,when an operator depresses a shutter button of the digital camera 11after adjustments of focus and exposure, the photography for the subjectP is carried out continuously twice. First, in the first photography,the flashlight emitting device 12 is in response to the instructionsignal transmitted from the digital camera 11 to emit flashlight ofhigher luminance than the illumination of the EL panel 130, so thatphotography for the subject P is carried out in the state of the followlight by the flashlight (step S102). Subsequently, the secondphotography is carried out in a state that the flashlight emittingdevice 12 turns off, that is, in the back light state by theillumination of only the EL panel 130 (step S103). Two photographicimages of the follow light photographic image and the back lightphotographic image, which are obtained through the step S102 and thestep S103, respectively, are temporarily stored in a memory of thedigital camera 11.

The processing of the step S101 to the step S103 as mentioned above isconcerned with photographic processing by person's hands such assalesclerks of a photo studio, which is carried out in the photo-studio10.

Next, there will be explained processing following step S104 of theflowchart of FIG. 2. Of the processing, which will be describedhereinafter, processing of the step S104 to the step S110 corresponds tothe first embodiment of a background replacing method of the presentinvention.

When the second photography is carried out through the processing of thestep S103 and the storage of the photographic image obtained through thesecond photography into the memory of the digital camera 11 iscompleted, two photographic images obtained through twice photography,which are temporarily stored in the memory, are immediately transferredto the personal computer 20. In step S104, when a customer operates thepersonal computer 20, a desired background is read out from the server40 or some input storage media. The obtaining processing for thephotographic images and the backgrounds, which are executed in the stepS104, corresponds to an example of the image obtaining step in thebackground replacing method of the present invention.

In step S105, the two photographic images are subjected to the imagecorrection processing including the correction of the trapezoidaldistortion, which will be described later.

In step S106, there is carried out a background replacing processing (itwill be described later), in which a background replaced image iscreated in accordance with the corrected photographic images. Here, inthe background replacing processing to be carried out in the step S106,examples of an area distinguishing process and a background replacingprocess in a background replacing method of the present invention areexecuted, respectively.

Next, image data representative of the background replaced image createdin the background replacing processing (the step S106) is firsttransferred to the image display unit 220 (step S107). In the imagedisplay unit 220, the background replaced image is displayed on thedisplay screen 220 a in accordance with the transferred image data (stepS108). On the display screen 220 a, there is displayed a messageinquiring of an operator of the personal computer 20 whether there is aneed to correct the background replaced image now displayed on thedisplay screen 220 a (step S109). In the event that a replay to themessage is indicative of a need of correction (“Yes” decision in thestep S109), an image correction, which will be described later, isapplied to the background replaced image (step S110). The imagecorrection to be executed in the step S110 corresponds to an example ofthe image correction process in the background replacing method of thepresent invention.

The processing of the step S107 to the step S110 is repeated until theoperator informs of no need of correction for the background replacedimage displayed on the display screen 220 a (“No” decision in the stepS109). According to the present embodiment, this processing makes itpossible to create finally natural background replaced image. Image datarepresentative of the finally created background replaced image istransferred to at least one of the printer 30 and an output storagemedia of a customer's wish (step S111). In the event that the image datais transferred to the printer 30, the printer 30 prints the backgroundreplaced image in accordance with the image data (step S112).

As to details of various sorts of processing in the embodiment of thebackground replacing method of the present invention, of the step S104to the step S110, they will be described in conjunction with functionsof respective sections of the background replacing apparatus of thepresent invention.

The processing of the step S104 to the step S110 including theembodiment of the background replacing method of the present inventionis executed in the personal computer 20. Hereinafter, there will bedescribed the personal computer 20 and details of the processing to beexecuted in the personal computer 20.

First, there will be described an internal structure of the personalcomputer 20.

FIG. 3 is a hardware structural view of the personal computer shown inFIG. 1.

The main frame unit 210 comprises, as shown in FIG. 3, a CPU 211 forexecuting a various types of program, a main memory 212 in which aprogram stored in a hard disk unit 213 is read out and developed forexecution by the CPU 211, the hard disk unit 213 for saving varioustypes of programs and data, a flexible disk (FD) drive 214 for accessinga flexible disk 520 mounted thereon, a CD-ROM drive 215 for accessing aCD-ROM 510 mounted thereon, an input interface 216 connected with theserver 40 and the digital camera 11 shown in FIG. 1 to receive replacingbackgrounds and photographic images from equipment such as the server 40and the digital camera 11, and an output interface 217 connected withthe printer 30 shown in FIG. 1 to output background replaced images.These various types of elements are connected via a bus 260 to the imagedisplay unit 220, the keyboard 230, the mouse 240, the media drive 250for accessing a small type of storage media 530.

The CD-ROM 510 corresponds to an embodiment of a background replacingprogram storage medium storing a background replacing program, of thepresent invention, which causes the personal computer 20 to operate asan embodiment of the background replacing apparatus of the presentinvention.

When the CD-ROM 510 is mounted on the CD-ROM drive 215, the programstored in the CD-ROM 510 is up-loaded on the personal computer 20 so asto be stored in the hard disk unit 213. When the background replacingprogram installed in the hard disk unit 213 is activated, the personalcomputer 20 serves as the background replacing apparatus of the presentinvention.

Next, there will be described an embodiment of a background replacingprogram related to the present invention, and an embodiment of abackground replacing program storage medium storing the backgroundreplacing program, of the present invention

FIG. 4 is a view useful for understanding an embodiment of a backgroundreplacing program storage medium of the present invention storing abackground replacing program related to the present invention.

The CD-ROM 510 stores therein a background replacing program 600, andcorresponds to an embodiment of a background replacing program storagemedium of the present invention.

The background replacing program 600 comprises an image obtainingsection 610, an image correcting section 620, a replacement processingsection 630, an image data transmission section 640, an image displaysection 650, and an image revision section 660.

As to details of every element of the background replacing program 600,they will be described in conjunction with functions of respectivesections of the background replacing apparatus of the present invention.In the following figures, the same reference numbers as those of FIG. 1,FIG. 3 and FIG. 4 denote the same parts.

FIG. 5 is a functional block diagram of elements for operating thepersonal computer shown in FIG. 1 and FIG. 3 in form of an embodiment ofa background replacing apparatus of the present invention, when thebackground replacing program shown in FIG. 4 is installed in thepersonal computer shown in FIG. 1 and FIG. 3. FIG. 5 also shows: thephoto-studio 10 comprising the digital camera 11, the flashlightemitting device 12, and the EL panel 130; the printer 30; and the server40.

According to a background replacing apparatus 700 shown in FIG. 5, thereis executed such background replacing processing that the digital camera11 of the photo-studio 10 is used to obtain a follow light photographicimage and a back light photographic image, which are photographed underthe follow light and the back light, and a subject and a background inthe follow light photographic image are distinguished from one anotherin accordance with the back light photographic image, so that thebackground is replaced by a desired background. The background replacingapparatus 700 comprises an image obtaining section 710, an imagecorrecting section 720, a replacement processing section 730, an imagedata transmission section 740, an image display section 750, and animage revision section 760.

The image obtaining section 710 first receives from the digital camera11 two photographic images, which are photographed by the digital camera11. The image obtaining section 710 reads a background of a customer'swish from the server 40 or an input storage media 540 such as CD-ROM andsmall type of storage media. The image obtaining section 710 issubstantially constructed in such a manner that the CPU 211 of thepersonal computer 20 controls the input interface 216, the FD drive 214,CD-ROM drive 215, and the media drive 250 in accordance with the imageobtaining section 610 of the background replacing program 600. The imageobtaining section 710 corresponds to an example of the image obtainingsection of the background replacing apparatus of the present invention.The image obtaining section 710 executes an example of the imageobtaining step in the background replacing method of the presentinvention.

The image correcting section 720 applies image correcting processingincluding the correction for the trapezoidal distortion, which will bedescribed later, to the two photographic images received by the imageobtaining section 710. The image correcting section 720 is substantiallyconstructed in such a manner that the CPU 211 of the personal computer20 operates in accordance with the image correcting section 620 of thebackground replacing program 600. As to details of the image correctingsection 720, it will be described later in conjunction with details ofthe image correcting processing with reference to other figures.

The replacement processing section 730 performs a background replacementprocessing in such a manner that a background replaced image is createdin accordance with the two photographic images subjected to the imagecorrecting processing by the image correcting section 720. Thereplacement processing section 730 is substantially constructed in sucha manner that the CPU 211 of the personal computer 20 operates inaccordance with the replacement processing section 630 of the backgroundreplacing program 600. As to details of the replacement processingsection 730, it will be described later in conjunction with details ofthe replacement processing with reference to other figures. Thereplacement processing section 730 corresponds to an example of oneserving both as the area discriminating section and the backgroundreplacing section of the background replacing apparatus of the presentinvention.

The image data transmission section 740 transmits image datarepresentative of the background replaced image created by thereplacement processing section 730 to at least one of the printer 30 andan output storage media 550 of a customer's wish, and an image displaysection 750 of the background replacing apparatus 700. The image datatransmission section 740 is substantially constructed in such a mannerthat the CPU 211 of the personal computer 20 controls the outputinterface 217, the FD drive 214, CD-ROM drive 215, and the media drive250 in accordance with the image data transmission section 640 of thebackground replacing program 600.

The image display section 750 displays on the display screen 220 a thebackground replaced image based on the image data transferred from theimage data transmission section 740. The image display section 750 issubstantially constructed in such a manner that the CPU 211 of thepersonal computer 20 controls the image display unit 220 in accordancewith the image display section 650 of the background replacing program600.

The image revision section 760 applies an image revision according to anoperator's instruction, which will be described later, to the backgroundreplaced image displayed on the display screen 220 a of the imagedisplay section 750. The image revision section 760 is substantiallyconstructed in such a manner that the CPU 211 of the personal computer20 operates in accordance with the image revision section 660 of thebackground replacing program 600. As to details of the image revisionsection 760, it will be described later with reference to other figures.The image revision section 760 corresponds to an example of the imagerevision section of the background replacing apparatus of the presentinvention. The image revision section 760 executes an example of theimage revision step in the background replacing method of the presentinvention.

In the event that the printer 30 receives from the image datatransmission section 740 the image data representative of the backgroundreplaced image that is finally completed, the printer 30 prints thebackground replaced image based on the image data. In the event that theoutput storage media 550 receives from the image data transmissionsection 740 the image data representative of the background replacedimage, the image data is written into the output storage media 550. As aresult, both the print sheet on which the background replaced image isprinted and the output storage media 550 into which the image datarepresentative of the background replaced image is written, or eitherone are provided in accordance with a customer's wish.

Next, hereinafter, there will be described details of the imagecorrecting section 720 and the replacement processing section 730 shownin FIG. 5.

First, there will be described the image correcting section 720.

FIG. 6 is an explanatory view useful for understanding details of theimage correcting section shown in FIG. 5. In the following figures, thesame reference numbers as those of FIG. 5 denotes the same parts.

The image correcting section 720 shown in FIG. 6 applies imagecorrecting processing including the correction for the trapezoidaldistortion to the follow light photographic image and the back lightphotographic image, which are received by the image obtaining section710. The image correcting section 720 comprises a marker detectionsection 721, a trapezoidal correction processing section 722, aneffective image creating section 723, and a color correcting section724.

Hereinafter, there will be described functions of every element of theimage correcting section 720 with reference to the follow lightphotographic image involving the trapezoidal distortion by way ofexample.

FIG. 7 is a typical illustration showing an example of a follow lightphotographic image involving a trapezoidal distortion.

FIG. 7 typically shows a photographic image involving a trapezoidaldistortion in which a width of an image is narrower as going down.

First, the marker detection section 721 detects four makers 131 cappearing on the photographic image. Essentially, coupling of those fourmakers 131 c ought to form a rectangle. However, according to theexample of FIG. 7, coupling of those four makers 131 c forms a trapezoidowing to the trapezoidal distortion.

The trapezoidal correction processing section 722 corrects a trapezoidaldistortion in such a manner that a suitable deformation is applied to aphotographic image in its entirety so that a figure, which is formedthrough coupling of the four makers 131 c detected by the markerdetection section 721, offers the essential rectangle.

FIG. 8 is a typical illustration showing a state that the trapezoidaldistortion of the follow light photographic image of FIG. 7 iscorrected.

As the photographic image to be subjected to the background replacingprocessing, from the point of view that a distinction between thesubject and the background can be easily made, it is preferable that thebackground is concerned with only the luminescent plane of the EL panel130. However, according to the present embodiment, it is intended tocontribute to saving space, and thus there is a limit in size of the ELpanel 130. Accordingly, as seen in FIG. 7 and FIG. 8, there appears onthe photographic image the unnecessary part other than the luminescentplane of the EL panel 130.

The effective image creating section 723 creates an effective imagewherein an image outside of an effective limit μl, which is defined fourmakers 131 c, is deleted from the photographic image that is subjectedto the correction processing by the trapezoidal correction processingsection 722.

FIG. 9 is a typical illustration showing an effective image that iscreated in accordance with the follow light photographic image correctedin the trapezoidal distortion, shown in FIG. 8.

After the effective image creating section 723 creates an effectiveimage wherein as shown in FIG. 9, the background of the subject P isconcerned with only the luminescent plane of the EL panel 130, the colorcorrecting section 724 applies to the effective image the colorcorrecting processing, for example, a red eye correction for person'seyes, and a correction of color shade of the whole image into apreferable color shade.

The above description is concerned with the explanation as to the imagecorrecting processing of the image correcting section 720 with referenceto the follow light photographic image of FIG. 7 to FIG. 9. According tothe present embodiment, the completely same processing as the imagecorrecting processing, which is applied to the follow light photographicimage, is applied also to the back light photographic image that isobtained together the follow light photographic image. However, in theback light photographic image, the subject appears with the completeblack. Accordingly, the color correcting section 724 omits theprocessing for the back light photographic image.

FIG. 10 is a typical illustration showing a corrected back lightphotographic image, which is subjected to image correction processing bythe image correcting section of FIG. 6.

As seen from FIG. 10, there is obtained the corrected back lightphotographic image, in which an area of the subject P is biased to theshadow side and an area of the EL panel 130 as the background is biasedto the highlight side. According to the follow light photographic imageof FIG. 7 to FIG. 9, luminance of the light emission of the EL panel 130is lower than luminance of the flashlight emitted from the flashlightemitting device 12. Therefore, while the luminescent plane of the ELpanel 130 is photographed more whitish than the actual luminescentcolor, the back light photographic image of FIG. 10 is photographed withindigo blue that is the actual luminescent color of the EL panel 130.

Thus, according to the corrected back light photographic image of FIG.10, the area of the subject P is quite different in brightness from thearea of the background. Therefore, it is possible to readilydiscriminate between the subject P and the background. Further,according to the corrected back light photographic image, the positionand the outline of the subject P in the image is substantially same asthose in the corrected follow light photographic image. Hence, accordingto the present embodiment, the replacement processing section 730 ofFIG. 5 distinguishes between the subject P and the background of thecorrected back light photographic image, and the distinction is appliedto the corrected follow light photographic image as it is. This featuremakes it possible to distinguish between the subject P and thebackground of the corrected follow light photographic image. Further,the replacement processing section 730 replaces the background of thefollow light photographic image by the background of a customer's wishto create a background replaced image.

Next, as to details of the replacement processing section 730, therewill be described the background replacement processing from thediscrimination between the subject and the background in the correctedback light photographic image up to the creation of the backgroundreplaced image.

As examples of the corrected follow light photographic image andcorrected back light photographic image, the following images will bereferred to.

FIG. 11 is an illustration showing an example of the corrected followlight photographic image. FIG. 12 is an illustration showing an exampleof the corrected back light photographic image.

FIG. 11 and FIG. 12 each show portraiture of a subject P′. Thephotographic image shown in FIG. 12 is one, which is photographedthrough the backlight. Thus, the area of the subject P′ is biased to theshadow side, and the area of the EL panel 130 as the background isbiased to the highlight side. Here, states of boundary periphery betweenthe subject P′ and the background will be shown with different figureswith respect to an area A1 of FIG. 11 and an area A2 of FIG. 12corresponding to the area A1 of FIG. 11.

FIG. 13 is an enlarged view of the area A1 shown in FIG. 11. FIG. 14 isan enlarged view of the area A1 shown in FIG. 12.

FIG. 13 and FIG. 14 show states that the EL panel 130 is seen throughhair of the subject P′. Generally, it is often seen in a photographicimage that there is a portion that a background appears through a partof the subject. Such a portion exists, as shown in the examples of FIG.11 to FIG. 14, at the boundary periphery between the subject and thebackground of the photographic image.

For the sake of convenience, FIG. 13 and FIG. 14 each show an idealstate that a hair of a person is visible up to one. According to theactual photographic image, however, in view of a restriction ofresolution, a pixel constituting an image is larger than the minutesubject such as a piece of hair. Hence, for example, an image of aportion that the EL panel 130 is seen through hair of the subject, asshown in FIG. 13 and FIG. 14, is constructed by a combination of pixelshaving a combination color in which a color of the minute subject suchas hair is combined with a color of a background that is seen throughthe minute subject. Therefore, on such a portion, it is impossible tosimply perform a distinction between the subject and the background.

In order to replace the background of the corrected follow lightphotographic image, as shown in FIG. 11 for instance, with a backgroundas a client wish, the replacement processing section 730 of FIG. 5 firstdiscriminates the corrected follow light photographic image among asubject area, a background area, and a boundary area interposed betweenthe subject area and the background area. The replacement processingsection 730 retains the subject area of the corrected follow lightphotographic image, and replaces the background area by anotherbackground of a customer's wish.

At that time, a portion that a background appears through a part of thesubject as shown in FIG. 13, that is, a portion constituted of pixelshaving a combination color in which a color of the minute subject and acolor of the background are combined, can not be simply discriminatedbetween the subject and the background, and thus the replacementprocessing section 730 discriminates that portion as the boundary area.

The replacement processing section 730 presumes, on the assumption thatthe whole pixels of the boundary area of the corrected follow lightphotographic image have the combination color as mentioned above, thesubject color that is the base of a combination of colors of the pixels,and the background color, and in addition presumes the combination ratioof the subject color and the background color. It is noted that theactual boundary area includes pixels having only the background colorand pixels having only the subject color as well as the pixels havingthe combination color. In the event that pixels of the processing objectare the pixels having only the background color, the replacementprocessing section 730 presumes a combination ratio in which a ratio ofthe subject color to the colors of the pixels takes “0”. In the eventthat pixels of the processing object are the pixels having only thesubject color, the replacement processing section 730 presumes acombination ratio in which a ratio of the subject color to the colors ofthe pixels takes “1”.

Thus, the replacement processing section 730 replaces the colors of thepixels of the boundary area by a combination color in which the presumedsubject color is combined with the different background color of thecustomer's wish at the presumed combination ratio. According to thisprocessing, the colors of the pixels having only the background color inthe boundary area is replaced with the different background color, andthe colors of the pixels having only the subject color is retained as itis, and the colors of the portion, in which the background appearsthrough a part of the subject, is replaced by the combination color. Asa result of the processing, even if the boundary area includes theportion that the background appears through a part of the subject, it ispossible to create a natural background replaced image in which adifferent background of a customer's wish appears through a part of thesubject.

Hereinafter, there will be described details of the replacementprocessing section 730.

FIG. 15 is an explanatory view useful for understanding details of thereplacement processing section shown in FIG. 5. In the followingfigures, the same reference numbers as those of FIG. 5 denotes the sameparts.

The replacement processing section 730 executes the above-mentionedbackground replacement processing. The replacement processing section730 comprises a mask creating section 731, a color presuming section732, and a background replaced image creating section 733. The maskcreating section 731 corresponds to an example of the areadiscriminating section of the background replacing apparatus of thepresent invention. A combination of the color presuming section 732 andthe background replaced image creating section 733 corresponds to anexample of the background replacing section of the background replacingapparatus of the present invention. The mask creating section 731executes an example of the area discriminating step of the backgroundreplacing method of the present invention. A combination of the colorpresuming section 732 and the background replaced image creating section733 executes an example of the background replacing step of thebackground replacing method of the present invention.

First, there will be explained outlines of those structural elements.

The mask creating section 731 creates, as to the corrected back lightphotographic image, a subject mask that defines a range in which onlypixels of the subject color exist, and a background mask that defines arange in which only pixels of the background color exist. Those twomasks are created with a gap in such a way that the pixels of theportion that the background appears through the subject, as shown inFIG. 14 for instance, are excluded from the ranges defined by thosemasks.

The color presuming section 732 applies the subject mask and thebackground mask, which are created by the mask creating section 731, tothe corrected follow light photographic image, and first, in the followlight photographic image, recognizes the gap provided between those twomasks, that is, the boundary area. The color presuming section 732presumes, on the assumption that the colors of the whole pixels in theboundary area are the mixed color of the subject color and backgroundcolor, the subject color that constitutes colors of the pixels, and thebackground color, and in addition presumes the combination ratio of thesubject color and the background color.

The background replaced image creating section 733 retains, first, as tothe corrected follow light photographic image, the area defined by thesubject mask, and replaces the area defined by the background mask withanother background of a customer's wish. Next, the background replacedimage creating section 733 determines, as to each pixel of the boundaryarea, a combination color through a combination of the color of thebackground of a customer's wish and the subject color presumed in thecolor presuming section 732 at the combination ratio presumed in thecolor presuming section 732, and replaces the color of the respectivepixel. This feature makes it possible to create a background replacedimage in which the background of the corrected follow light photographicimage is naturally replaced with another background of a customer'swish.

The image data transmission section 740 of FIG. 5 transmits image datarepresentative of the background replaced image thus created to outputdevices and the output storage media 550 of FIG. 5.

Next, there will be explained details of the respective structuralelements.

First, there will be explained details of the mask creating section 731.

FIG. 16 is an explanatory view useful for understanding details of themask creating section shown in FIG. 15.

The mask creating section 731 comprises a histogram creating section 731a, a threshold computing section 731 b, a subject mask set up section731 c, a background mask set up section 731 d, a subject mask reductionsection 731 e, and a background mask reduction section 731 f.

The histogram creating section 731 a creates histograms as to luminousintensity of pixels constituting the corrected back light photographicimage.

FIG. 17 is a view showing various examples of histograms as to luminousintensity of pixels constituting the corrected back light photographicimage.

A part (a) of FIG. 17 shows a luminance histogram H1 in which luminanceof pixels constituting the back light photographic image is divided intoa large number of luminance ranges, and the number of pixels, which isdivided into the associated luminance range, is expressed by frequency.In the luminance histogram H1, the horizontal axis H1_1 denotesluminance, and the vertical axis H1_2 denotes frequency.

According to the present embodiment, as to the follow light photographicimage and the back light photographic image, colors of the pixels areexpressed by coordinates on an RGB color space defining colors withthree colors of R (red), G (green), and B (blue), that is, R-value,G-value, and B-value in the RGB color space. The R-value, the G-value,and the B-value, which are representative of colors of the pixels,reflect luminous intensity of their associated pixel. Thus, it ispossible to create a histogram as to luminous intensity of the pixelsconstituting the back light photographic image using anyone of theR-value, the G-value, and the B-value.

A part (b) of FIG. 17 shows an R-value histogram H2 in which R-values ofpixels constituting the back light photographic image are divided into alarge number of ranges of R-values, and the number of pixels, which isdivided into the associated range, is expressed by frequency. A part (c)of FIG. 17 shows a G-value histogram H3 in which G-values of pixelsconstituting the back light photographic image are divided into a largenumber of ranges of G-values, and the number of pixels, which is dividedinto the associated range, is expressed by frequency. A part (d) of FIG.17 shows a B-value histogram H4 in which B-values of pixels constitutingthe back light photographic image are divided into a large number ofranges of B-values, and the number of pixels, which is divided into theassociated range, is expressed by frequency. In the R-value histogramH2, the horizontal axis H2_1 denotes R-value, and the vertical axis H2_2denotes frequency. In the G-value histogram H3, the horizontal axis H3_1denotes G-value, and the vertical axis H3_2 denotes frequency. In theB-value histogram H4, the horizontal axis H4_1 denotes B-value, and thevertical axis H4_2 denotes frequency.

According to the present embodiment, with respect to the back lightphotographic image, the background of the subject appears with indigoblue that is the luminescent color of the EL panel 130. Thus, theG-value, of the R-value, the G-value and the B-value, most effectivelyreflects the luminous intensity of the pixels of the back lightphotographic image. Accordingly, the histogram creating section 731 a ofFIG. 16 is constructed so as to create the G-value histogram H3, whichis shown in the part (c) of FIG. 17, of four histograms shown in FIG.17.

In the G-value histogram H3, in a similar fashion to other threehistograms, there appear two peaks Pk1 and Pk2 at the shadow side andthe highlight side, respectively. The peak Pk1 of the shadow sideappears through the division of the pixels of the subject area in thecorrected back light photographic image. The peak Pk2 of the highlightside appears through the division of the pixels of the background areain the corrected back light photographic image.

The threshold computing section 731 b shown in FIG. 16 computes twothresholds, as set forth below, in accordance with the two peaks Pk1 andPk2 in the G-value histogram H3 created by the histogram creatingsection 731 a of FIG. 16, respectively. As shown in the part (c) of FIG.17, there is computed from the peak Pk1 of the shadow side a firstthreshold Sr1, which is the upper limit of the G-value where thefrequency offers a predetermined value or more on the peak Pk1. Andthere is computed from the peak Pk2 of the highlight side a secondthreshold Sr2, which is the lower limit of the G-value where thefrequency offers a predetermined value or more on the peak Pk2.

The subject mask set up section 731 c shown in FIG. 16 sets up as aninitial subject mask an area consisting of pixels where the G-valueoffers the first threshold Sr1 or less, of a plurality of pixelsconstituting the corrected back light photographic image. The backgroundmask set up section 731 d sets up as an initial background mask an areaconsisting of pixels where the G-value offers the second threshold Sr2or more.

Thus, according to the present embodiment, there are set up two types ofmasks in accordance with two peaks Pk1 and Pk2 in the G-value histogramH3 created in histogram creating section 731 a, respectively. In orderto set up the two types of masks with great accuracy, it is desired thatthe two peaks Pk1 and Pk2 are sharp as much as possible. However,different from the present embodiment, for example, in a state that aphoto studio is illuminated with a steady illumination, it may happenthat a back light photographic image becomes a light image in itsentirety. This brings about a possibility that it is difficult to obtaintwo sharp peaks in the G-value histogram.

Hereinafter, there will be explained a histogram capable of obtainingtwo sharp peaks in the light photo studio too, which is different fromthe histogram created in accordance with the present embodiment.

FIG. 18 is a view showing an example of a histogram different from theexample of the histogram shown in FIG. 17.

A histogram H5 shown in FIG. 18 is created using both the correctedfollow light photographic image and the corrected back lightphotographic image, different from the example of the histogram shown inFIG. 17, in the manner as set forth below.

First, on all sets of pixels of the associated positions between thosetwo corrected photographic images, there are computed ratios ofluminance among sets of pixels. Next, the computed ratios of luminanceamong sets of pixels are divided into a large number of ranges. Thus,the histogram H5 is created in such a way that the number of dividedsets on the individual range is expressed by frequency. In the histogramH5, a horizontal axis H5_1 denotes a ratio of luminance and a verticalaxis H5_2 denotes frequency. The ratios of luminance among sets ofpixels correspond to an example of “ratios of brightness among aplurality of photographic images” referred to in the present invention.

The ratios of luminance that is computed in accordance with a followlight photographic image and a back light photographic image, which areobtained through photography in a photo studio, are expressed by thefollowing equations.

N1=(Ri×Lt)/(Ri×(Ls+Lt))=Lt/(Ls+Lt)  (1)

N2=(Ri×Lt+Le)/(Ri×(Ls+Lt)+Le)  (2)

Where Ls denotes luminance of flash of a flash light-emitting apparatusin the photo studio, Lt denotes luminance of light illumination thatexists steadily in the photo studio, Ri denotes reflection factors forpositions corresponding to respective pixels of a photographic image onthe subject or the EL panel, Le denotes luminance of emitted light bythe EL panel, N1 denotes a ratio of luminance on pixels corresponding tothe subject area of the photographic image, and N2 denotes a ratio ofluminance on pixels corresponding to the background area of thephotographic image.

From the equation (1), it is understood that the ratio of luminance onthe subject area offers the constant value, which is determined inaccordance with the specifications of the independent apparatus in thephoto studio, independent from the position in the subject area.

In the equation (2), Ri is the reflection factors on the EL panel. Thereflection factors offer constant value on the EL panel independently ofthe position. Thus, from the equation (2), it is understood that theratio of luminance on the background area also offers the constantvalue, which is determined in accordance with the specifications of theindependent apparatus in the photo studio, independent from the positionin the background area.

However, between the ratio of luminance on the subject area and theratio of luminance on the background area, there is a difference owingto the luminance (Le) of emitted light by the EL panel. As a result, asshown in FIG. 18, on the histogram H5 as to the ratio, there appear twosharp peaks Pk3 and Pk4, which correspond to the subject area and thebackground area, respectively. Between the follow light photographicimage and the back light photographic image, a difference in luminanceon the subject area is larger than a difference in luminance on thebackground area. Thus, in the histogram H5 shown in FIG. 18, the peakPk3 at the low ratio side corresponds to the background area, and thepeak Pk4 at the high ratio side corresponds to the subject area.

In this manner, in the histogram H5 shown in FIG. 18, even if lightillumination exists steadily in the photo studio, two sharp peaks alwaysappear. Further, even if the luminance of the steady illuminationvaries, it would be understood from the equations (1) and (2) that thesharpness of the peaks does not vary, while the positions of the peaksvary.

The above description is concerned with the explanation of the histogramof the ration of the luminance. Now, let us return to the explanation ofthe image taking system 1 of the present embodiment.

As mentioned above, according to the image taking system 1 of thepresent embodiment, the initial subject mask and the initial backgroundmask are set up in accordance with two peaks Pk1 and Pk2 in the G-valuehistogram H3 shown in part (c) of FIG. 17, respectively.

FIG. 19 is an illustration showing an area that is set up as an initialsubject mask and an area that is set up as an initial background mask,in the enlarged view of the back light photographic image shown in FIG.14.

A hatching portion of the right side in FIG. 19 is a portion of hairs ofthe subject P′ in the enlarged view of the back light photographic imageshown in FIG. 14, in which the hairs are so concentrated that thebackground is not seen, and are greatly biased to the shadow sidethrough the back light photography. The G-value of such a portion isless than the first threshold Sr1. As a result, the subject mask set upsection 731 c sets up the hatching portion of the right side in FIG. 19as an initial subject mask M1.

A hatching portion of the left side in FIG. 19 is a portion of only theEL panel 130 of the background in the enlarged view of the back lightphotographic image shown in FIG. 14, in which the background is greatlybiased to the highlight side. The G-value of such a portion exceeds thesecond threshold Sr2. As a result, the background mask set up section731 d sets up the hatching portion of the left side in FIG. 19 as aninitial background mask M2.

On the other hand, a portion, which belongs to none of the hatchingportions of the left side and the right side in FIG. 19, is a portionthat the background appears through the hairs. The G-value of such aportion exceeds the first threshold Sr1 and is less than the secondthreshold Sr2. As a result, the portion, which belongs to none of thehatching portions of the left side and the right side in FIG. 19, islocated between the initial subject mask M1 and the initial backgroundmask M2, and remains in form of a boundary area M3.

FIG. 20 is an illustration showing the initial subject mask and theinitial background mask shown in FIG. 19. FIG. 21 is an illustrationshowing a state that the initial subject mask and the initial backgroundmask are overlapped on the corrected follow light photographic imageshown in FIG. 11 in its entirety.

As shown in FIG. 20 and FIG. 21, the initial subject mask M1 is close tothe initial background mask M2, so that the boundary area M3 isnarrowed. As mentioned above, regarding the corrected back lightphotographic image, the position and outline of the subject in the imageare substantially same as those of the corrected follow lightphotographic image. However, there is a possibility of an occurrence ofsomewhat discrepancy in position and outline of the subject between thefollow light photographic image and the back light photographic image,owing to somewhat discrepancy in timing of the photography between thefollow light photographic image and the back light photographic image.In such a case, the narrow boundary area M3 may involve such apossibility that regarding the corrected follow light photographic imageshown, the portion that the background appears through a part of thesubject, as shown in FIG. 13, is not completely included in the boundaryarea M3, and thus is included either one of the initial subject mask M1and the initial background mask M2. In the event that the replacement ofthe background is performed in accordance with the initial subject maskM1 and the initial background mask M2 as mentioned above, there is apossibility of an occurrence of problems that the old background remainson the portion that the background appears through a part of thesubject, or that portion is completely replaced by the new background,so that the subject has a chip in a part.

In order to avoid those problems, according to the present embodiment,the subject mask reduction section 731 e shown in FIG. 16 reduces theinitial subject mask M1 in a direction going away from the boundary areaM3, and the background mask reduction section 731 f reduces the initialbackground mask M2 in a direction going away from the boundary area M3.This feature makes it possible to expand the boundary area M3 so as toinclude much the portion that the background appears through a part ofthe subject.

FIG. 22 is an illustration showing a state that the initial subject maskand the initial background mask shown in FIG. 20 are reduced in adirection going away from the boundary area, respectively.

As seen from FIG. 22, the initial subject mask M1 is reduced in such away that an outline L1 of the initial subject mask M1 is moved by thecorresponding predetermined number of pixels in a direction going awayfrom the boundary area M3. In a similar fashion, the initial backgroundmask M2 is reduced in such a way that an outline L2 of the initialbackground mask M2 is moved by the corresponding predetermined number ofpixels in a direction going away from the boundary area M3. According tothe present embodiment, as a migration length of the outlines of twomasks, there is adopted 5 pixels. Final subject mask M1′ and finalbackground mask M2′ are completed through reduction of the masks asmentioned above.

FIG. 23 is an illustration showing a state that the subject mask and thebackground mask, which are finally completed, are overlapped on thecorrected follow light photographic image shown in FIG. 11 in itsentirety.

A gap between the subject mask M1′ and the background mask M2′, whichare finally completed, that is, a final boundary area M3′ is expanded bythe corresponding 10 pixels as compared with the boundary area M3 shownin FIG. 21. As a result, as seen from FIG. 23, the final boundary areaM3′ includes much the portion that the background appears through a partof the subject. An area defined by the background mask M2′, which isfinally completed, corresponds to an example of the “background area ofthe photographic image” referred to in the resent invention. Acombination of an area defined by the subject mask M1′, which is finallycompleted, and the final boundary area M3′ corresponds to an example of“another area excepting the background area of the photographic image”.

The above-description is concerned with the description of the maskcreating section 731 of the replacement processing section 730 shown inFIG. 15. Next, before explanation of details of the color presumingsection 732 of the replacement processing section 730, there will beexplained a mask creating section for creating a background mask and asubject mask in accordance with a method different from the presentembodiment, which is different from the mask creating section 731.

For example, in the photo-studio 10 shown in FIG. 1, it would happenthat a relative positional relation between the EL panel 130 and thedigital camera 11 may cause the EL panel 130 to come out darker in thephotographic image at the lower portion near feet of the subject ascompared with the upper portion, brightness of the corrected back lightphotographic image, which will be used for creation of the mask, isuneven in a height direction. In such a case, it happens that the peakcorresponding to the background area of the histogram shown in FIG. 17is dispersed, and it is difficult to create a background mask with greataccuracy. According to a mask creating section, which is different fromthe mask creating section 731 of the present embodiment, it is possibleto cope with the problem as mentioned above. Hereinafter, there will bedescribed such a mask creating section.

FIG. 24 is a view showing a mask creating section, which is differentfrom the mask creating section 731 of the present embodiment.

A mask creating section 731′, which is different from the mask creatingsection 731, comprises an image dividing section 731 a′, a histogramcreating section 731 b′, a threshold computing section 731 c′, a subjectmask set up section 731 d′, a background mask set up section 731 e′, asubject mask reduction section 731 f′, a background mask reductionsection 731 g′, a subject mask combining section 731 h′, and abackground mask combining section 731 i′.

The image dividing section 731 a′ divides the corrected back lightphotographic image in the manner as set forth below.

FIG. 25 is a view showing a state that the corrected back lightphotographic image is divided.

FIG. 25 shows a state in which the corrected back light photographicimage is divided into three in a height direction. As mentioned above,unevenness of the corrected back light photographic image in luminousintensity appears in the height direction. Accordingly, in individualinsides of three small areas Ar1, Ar2, and Ar3, which are obtainedthrough the division in the height direction, the luminous intensity isrelatively even.

The histogram creating section 731 b′ shown in FIG. 24 createshistograms on individual three small areas Ar1, Ar2, and Ar3.

FIG. 26 is a view showing examples of the histograms, which are createdon the three small areas Ar1, Ar2, and Ar3 shown in FIG. 25,respectively.

A part (a) of FIG. 26 shows a histogram H6 of G-value associated withthe small areas Ar1 of the highest stage in the FIG. 25. A part (b) ofFIG. 26 shows a histogram H7 of G-value associated with the small areasAr2 of the middle stage in the FIG. 25. A part (c) of FIG. 26 shows ahistogram H8 of G-value associated with the small areas Ar3 of thelowest stage in the FIG. 25. As shown in FIG. 26, in the histograms,which are created on the three small areas Ar1, Ar2, and Ar3, the peaksassociated with the background areas appear with sharpness capable ofbeing discriminated at the highlight side.

The threshold computing section 731 c′ shown in FIG. 24 computes athreshold for distinguish the subject area from other area and athreshold for distinguish the background area from other area, asmentioned above, on three histograms shown in FIG. 26.

The subject mask set up section 731 d′ and the background mask set upsection 731 e′ perform set up of the initial subject mask and theinitial background mask as mentioned above, using the thresholdscomputed in the threshold computing section 731 c′, on the three smallareas Ar1, Ar2, and Ar3, respectively. The subject mask reductionsection 731 f′ and the background mask reduction section 731 g′ reduceinitial subject mask and the initial background mask set up on the threesmall areas Ar1, Ar2, and Ar3, respectively, to create the subject maskand the background mask as mentioned above.

The subject mask combining section 731 h′ combines the subject masks,which are created on the three small areas Ar1, Ar2, and Ar3,respectively, to complete the final subject mask associated with theback light photographic image in its entirety. In a similar fashion, thebackground mask combining section 731 i′ combines the background masks,which are created on the three small areas Ar1, Ar2, and Ar3,respectively, to complete the final background mask associated with theback light photographic image in its entirety.

According to the mask creating section 731′ as mentioned above, thecorrected back light photographic image is divided into small areas,which are relatively even in luminous intensity, and there are createdsubject masks and the background masks on the small areas. The createdsubject masks and the background masks are combined to complete thefinal subject masks and the background masks. This processing makes itpossible to create a mask with great accuracy, even if the correctedback light photographic image to be used for creating the masks isuneven in luminous intensity in the height direction. According to thepresent embodiment, as the number of divisions where the corrected backlight photographic image is divided into small areas, there are raisedthree divisions by way of examples. However, the number of divisions isnot restricted to three.

Here, the explanation of the mask creating section 731′ will beterminated, and let us return to the explanation of the replacementprocessing section 730 shown in FIG. 15. First, there will be explaineddetails of the color presuming section 732 of the replacement processingsection 730.

FIG. 27 is an explanatory view useful for understanding details of thecolor presuming section shown in FIG. 15.

The color presuming section 732 comprises a background color presumingsection 732 a, a subject color presuming section 732 b, and acombination ratio presuming section 732 c.

First, there will be described the background color presuming section732 a.

FIG. 28 is an explanatory view useful for understanding details of thebackground color presuming section shown in FIG. 27.

The background color presuming section 732 a presumes, as to therespective pixels of the final boundary area in the corrected followlight photographic image, a background color constituting a color of thepixel. The background color presuming section 732 a comprises aretrieval range set up section 732 a_1, a background area confirmationsection 732 a_2, and a background color computing section 732 a_3.

In order to presume a background color on a certain pixel in theboundary area, the background color presuming section 732 a searches,from among pixels in the area defined by the background mask, aplurality of pixels located near the pixels in some extent, anddetermines an average color of colors of the plurality of pixels thusretrieved as a background color on the pixel.

The retrieval range set up section 732 a_1 sets up a square-like shapedretrieval range on the pixel of presuming the background as follows.

FIG. 29 is an illustration useful for understanding a state that when abackground color is estimated on a certain pixel, a search range is setup on the pixel.

In order to set up a retrieval range on a certain pixel G in theboundary area, the retrieval range set up section 732 a_1 graduallyexpands a retrieval range starting from a square-like shapedpredetermined initial range T1 on the pixel G. The retrieval range setup section 732 a_1 stops the expansion when the pixels in the areadefined by the background mask M2′ exceeds a predetermined number ofpixels and are included in the retrieval range, and sets up a retrievalrange T1′ at that time as a retrieval range for the backgroundpresumption.

When the retrieval range set up section 732 a_1 sets up the retrievalrange T1′, the background area confirmation section 732 a_2 shown inFIG. 24 confirms the subsequent area in the corrected follow lightphotographic image as an area used for presuming the background color onthe pixel G.

FIG. 30 is an illustration showing an example of an area used forpresuming a background color on a certain pixel.

As seen from FIG. 30, as the area used for presuming the backgroundcolor on the pixel G, there is confirmed an area B1 in the correctedfollow light photographic image, which is the area defined by thebackground area, and is in the retrieval range T1′ set up by theretrieval range set up section 732 a_1.

Thus, when the area B1 used for presuming the background color on thepixel G is confirmed, the background color computing section 732 a_3shown in FIG. 28 computes the average color of a plurality of pixels inthe area B1, that is, the respective average values of R-values,G-values, and B-values of the plurality of pixels. A color representedby the computing result is adopted as the background color on the pixelG.

The background color presuming section 732 a performs the processingfrom the set up of the retrieval range to the computing of the averagecolor on all the pixels in the boundary area of the corrected followlight photographic image.

Next, there will be explained the subject color presuming section 732 bshown in FIG. 27.

FIG. 31 is an explanatory view useful for understanding details of thesubject color presuming section shown in FIG. 27.

The subject color presuming section 732 b presumes, as to the respectivepixels of the final boundary area in the corrected follow lightphotographic image, a subject color constituting a color of the pixel.The subject color presuming section 732 b comprises a retrieval rangeset up section 732 b_1, a subject area confirmation section 732 b_2, apixel color confirmation section 732 b_3 and a subject color retrievalsection 732 b_4.

In order to presume a subject color on a certain pixel in the boundaryarea, the subject color presuming section 732 b searches, from amongpixels in the area defined by the subject mask, a plurality of pixelslocated near the pixels in some extent, and searches a subject color onthe pixel from among the plurality of pixels thus retrieved inaccordance with a retrieval method which will be described later.

The retrieval range set up section 732 b_1 sets up a square-like shapedretrieval range on the pixel of presuming the subject as follows.

FIG. 32 is an illustration useful for understanding a state that when asubject color is presumed on a certain pixel, a search range is set upon the pixel.

In order to set up a retrieval range on a certain pixel G in theboundary area, the retrieval range set up section 732 b_1 graduallyexpands a retrieval range starting from a square-like shapedpredetermined initial range T2 on the pixel G. The retrieval range setup section 732 b_1 stops the expansion when the pixels in the areadefined by the subject mask M1′ exceeds a predetermined number of pixelsand are included in the retrieval range, and sets up a retrieval rangeT2′ at that time as a retrieval range for the subject presumption.

When the retrieval range set up section 732 b_1 sets up the retrievalrange T2′, the subject area confirmation section 732 b_2 shown in FIG.31 confirms the subsequent area in the corrected follow lightphotographic image as an area used for presuming the subject color onthe pixel G.

FIG. 33 is an illustration showing an example of an area used forpresuming a subject color on a certain pixel.

As seen from FIG. 33, as the area used for presuming the subject coloron the pixel G, there is confirmed an area B2 in the corrected followlight photographic image, which is the area defined by the subject area,and is in the retrieval range T2′ set up by the retrieval range set upsection 732 b_1.

Thus, when the area B2 used for presuming the subject color on the pixelG is confirmed, the pixel color confirmation section 732 b_3 shown inFIG. 31 confirms a color of the pixel G, and then the subject colorretrieval section 732 b_4 shown in FIG. 31 retrieves the subject coloron the pixel G from among a plurality of pixels in the area B2, inaccordance with a retrieval method as set forth below.

FIG. 34 is an illustration useful for understanding a method ofsearching a subject color on a certain pixel in a boundary area of thecorrected follow light photographic image.

FIG. 34 shows, as to a certain pixel in a boundary area of the correctedfollow light photographic image, a straight line CL1 coupling abackground color C1 presumed by the background color presuming section732 a shown in FIG. 27 with a color C2 of the pixel, which is confirmedby the pixel color confirmation section 732 b_3 shown in FIG. 31, on theRGB color space.

As mentioned above, as to the color of the pixel in the boundary area ofthe corrected follow light photographic image, there is a possibilitythat a background appears through a part of the subject at the time ofphotography, and the background color is combined with the subject colorinto a combination color. In the event that the color of the pixel to beretrieved in the subject color is such a combination color, the color C2of the pixel, which is the combination color, the background color C1 asthe base of the color C2, and the subject color are lined up on the RGBcolor space.

Thus, the subject color retrieval section 732 b_4 shown in FIG. 31 firstretrieves, from among colors of a plurality of pixels in the area B2, acolor C3 lined up on the extended line of the straight line CL1.

In the event that there is a plurality of colors C3, there may beexpected the highest possibility that the color closest to the color C2of the pixel, on the extended line on the straight line CL1, is the baseof the color C2 as well as the background color C1. Hence, in the eventthat there exists a plurality of colors C3 lined up on the extended lineof the straight line CL1, the subject color retrieval section 732 b_4adopts the color closest to the color C2 of the pixel, from among thosecolors C3, as a subject color C4 on the pixel.

The subject color presuming section 732 b performs the processing fromthe set up of the retrieval range to the retrieval of the subject coloron all the pixels in the boundary area of the corrected follow lightphotographic image.

The above description is concerned with the explanation of the subjectcolor presuming section 732 b shown in FIG. 27. Next, there will beexplained the combination ratio presuming section 732 c shown in FIG. 27with reference to FIG. 34.

The combination ratio presuming section 732 c computes, as shown in FIG.34, the ratio of the length of a line segment coupling on the RGB colorspace the background color C1 presumed by the background color presumingsection 732 a with the subject color C4 presumed by the subject colorpresuming section 732 b to the length of a line segment coupling thebackground color C1 with the color C2 of the pixel, in form of acombination ratio α where the color C2 of the pixel is combined with thebackground color C1 and the subject color C4. The combination ratiopresuming section 732 c performs the computation of the combinationratio α on all the pixels in the boundary area of the corrected followlight photographic image.

The above description is concerned with the explanation of details ofthe color presuming section 732 of the replacement processing section730 shown in FIG. 15. Next, there will be explained details of thebackground replaced image creating section 733 of the replacementprocessing section 730.

The background replaced image creating section 733 retains, as mentionedabove, first, as to the corrected follow light photographic image, thearea defined by the subject mask, and replaces the area defined by thebackground mask with another background of a customer's wish.

Next, the background replaced image creating section 733 determines, asto each pixel of the boundary area, a combination color through acombination of the color of the background of a customer's wish and thesubject color presumed in the color presuming section 732 at thecombination ratio presumed in the color presuming section 732.

FIG. 35 is an illustration useful for understanding a scheme of acombination color on a certain pixel in a boundary area of the correctedfollow light photographic image.

FIG. 35 shows a straight line CL1 coupling among the presumed backgroundcolor C1, the color C2 of the pixel, and the presumed subject color C4,on the RGB color space, which is also shown in FIG. 34. The color C2 ofthe pixel is a point wherein a straight line coupling the backgroundcolor C1 with the subject color C4 is divided in a ratio of thecombination ratio α to (1−the combination ratio α). Thus, the backgroundreplaced image creating section 733 adopts, as shown in FIG. 35, a pointwherein a straight line CL2 coupling another background color C5 withthe presumed subject color C4 is divided in a ratio of the combinationratio α to (1−the combination ratio α), in form of a combination colorC6.

When the combination color is determined on all the pixels of theboundary area in accordance with the method explained with reference toFIG. 35, the background replaced image creating section 733 replaces thecolor of all the pixels with the combination color determined on therespective pixel. As a result, there is completed a background replacedimage in which the background of the corrected follow photographic imageis replaced by another background of a customer's wish.

As seen from FIG. 23, the boundary area of the corrected followphotographic image includes the portion occupied by only the subject andthe portion occupied by only the background as well as the portion thatthe background appears through a part of the subject. The color of thepixels of the portion occupied by only the subject is the color of thesubject per se. Thus, even if the background is replaced by anotherbackground, there is a need to be maintained in color of the pixel. Thecolor of the pixels of the portion occupied by only the background isthe color of the background per se. Thus, when the background isreplaced by another background, there is a need that a color of thepixel is completely replaced by the other background color.

According to the present embodiment, in the event that the pixels of thepresuming object of the background color, the subject color and thecombination ratio are concerned with pixels of the portion occupied byonly the subject, a color of all most of the pixels in an area used forpresuming the subject color on the associated pixel, such as the area B2shown in FIG. 33, is the color of the subject per se that is the same asthe color of the pixel of the presuming object. As a result, as to sucha pixel, the combination ratio α offers approximately “1”. When thebackground is replaced, the combination color is determined inaccordance with the combination ratio α. Thus, it is possible tomaintain the color of the subject per se, which is the color of theassociated pixel.

On the other hand, in the event that the pixels of the presuming objectof the background color, the subject color and the combination ratio areconcerned with pixels of the portion occupied by only the background,the color of the associated pixel is the color of the background per sethat is the same as the background color to be presumed on the pixel. Asa result, as to such a pixel, the combination ratio α offersapproximately “0”. When the background is replaced, the color of theassociated pixel is completely replaced with another background color inaccordance with the combination ratio α.

In effect, according to the present embodiment, in the event that thepixels of the presuming object of the background color, the subjectcolor and the combination ratio are concerned with pixels of the portionthat the background appears through a part of the subject, thecombination ratio α has a values between “0” and “1”, and thus when thebackground is replaced, the subject color is combined with anotherbackground color to create a background replaced image.

FIG. 36 is a view showing an example of a background replaced image.

FIG. 36 shows an enlarged view in which the same portion as the area A1shown in FIG. 11 is enlarged, in a background replaced image createdthrough a replacement of the background of the corrected follow lightphotographic image shown in FIG. 11 by the different background.

The background replaced image created in the background replaced imagecreating section 733 is an image in which a color of the pixels in theboundary area is replaced by the combination color. Thus, as to thebackground replaced image, a person recognizes, as shown in FIG. 36, astate that a replaced different background TB is seen through hairs ofthe subject P′.

Now to repeatedly say, according to the present embodiment, the initialsubject mask M1 and the initial background mask M2, which are created inaccordance with the corrected back light photographic image, are reducedby the subject mask reduction section 731 e and the background maskreduction section 731 f shown in FIG. 16, respectively, by thecorresponding 5 pixels in a direction going away from the boundary areaso as to complete the subject mask M1′ and the background mask M2′. Thisfeature contributes to avoiding such the inconveniences that a portionthat the background appears through a part of the subject in thephotographic image is out of the boundary area, owing to the discrepancyin position and outline of the subject, which may occur between thecorrected follow light photographic image and the corrected back lightphotographic image. However, in some situation of the subject at thetime of photography, there is a case that the reduction by thecorresponding 5 pixels is insufficient to compensate for the discrepancyin position and outline of the subject. In such a case, there will beraised problems that the old background remains at the portion that thebackground appears through a part of the subject, in the backgroundreplaced image displayed on the display screen 220 a (cf. FIG. 1), orthat the portion is replaced by the new background in its entirety, sothat a part of the subject is chipped.

According to the present embodiment, in order to correct such unnaturalportions of the background replaced image and obtain a finally naturalbackground replaced image, there is provided the image revision section760 (cf. FIG. 5).

Hereinafter, there will be explained details of the image revisionsection 760.

FIG. 37 is an explanatory view useful for understanding details of theimage revision section 760 shown in FIG. 5. In the following figures,the same reference numbers as those of FIG. 1 and FIG. 5 denote the sameparts.

FIG. 37 shows the subject mask reduction section 731 e and thebackground mask reduction section 731 f also shown in FIG. 16 as well asthe image revision section 760.

The image revision section 760 shown in FIG. 37 revises the unnaturalportion of the background replaced image displayed on the display screen220 a. The image revision section 760 comprises a revision receivingsection 761, a revision value input section 762, and a re-replacementinstruction section 763.

As mentioned above, there are displayed, on the display screen 220 a, amessage inquiring as to whether there is a necessity for revision on thebackground replaced image, as well as the background replaced image. Atthat time, when an operator, who looks at the unnatural portion in thebackground replaced image, operates the keyboard 230 and the mouse 240to indicate that there is a necessity for revision on the backgroundreplaced image, the revision receiving section 761 receives theoperator's instruction.

When the revision receiving section 761 receives the operator'sinstruction for revision, the revision value input section 762 causesthe display screen 220 a to display an input image plane for inputtingrevision values as set forth below.

According to the present embodiment, instead of the reduction amount ofthe corresponding 5 pixels to the respective masks of the subject maskreduction section 731 e and the background mask reduction section 731 falso shown in FIG. 16, a new reduction amount is treated as the revisionvalue, and this new reduction amount is set up to the value of 5 pixelsor more.

This reduction amount is entered on the input image plane for revisionvalues, which is displayed by the revision value input section 762,through operation of the keyboard 230 and the mouse 240 by an operator.The revision value input section 762 transfers the entered reductionamount to the subject mask reduction section 731 e and the backgroundmask reduction section 731 f. The subject mask reduction section 731 eand the background mask reduction section 731 f reduce over again theinitial subject mask M1 and the initial background mask M2 with thereduction amount transferred from the revision value input section 762instead of 5 pixels, respectively. Thus, there are created new subjectmask and background mask, which are narrower in width as compared withthose concerned with the reduction amount of 5 pixels. As a result, theboundary area is expanded, so that the portion that the backgroundappears through a part of the subject is much included in the boundaryarea as compared with a case where the reduction amount is 5 pixels.

The re-replacement instruction section 763 issues instructions ofre-replacement processing for the background using the new subject maskand background mask to the respective portions of the replacementprocessing section 730.

The new background replaced image, which is created through thereplacement processing, is displayed on the display screen 220 a. Theimage revision by the image revision section 760 is repeated until theunnatural portion disappears from the background replaced imagedisplayed on the display screen 220 a.

As explained with reference to FIG. 1 to FIG. 37, according to the imagetaking system 1 of the present embodiment, first, the background and thesubject in the corrected follow light photographic image aredistinguished from one another in accordance with the corrected backlight photographic image in which the background and the subject of theimage are greatly different from one another in luminous intensity, sothat the background and the subject of the image are distinguishablefrom one another without making a mistake. Hence, according to the imagetaking system 1 of the present embodiment, it is possible to suppresssuch an inconvenience that when the background of the corrected followlight photographic image is replaced by a different background, a partof the subject is erroneously replaced by a background, or an oldbackground remains in the background replaced image, thereby creating abackground replaced image in which the background is naturally replacedby a background of a client wish.

Further, according to the image taking system 1 of the presentembodiment there is recognized a boundary area between the subject areaand the background area in the corrected follow light photographicimage. When the background is replaced, the color of the pixel in theboundary area is replaced by a combination color in which the differentbackground color to be replaced is suitably combined with the subjectcolor. Thus, even if there is such a portion that the background appearsthrough a part of the subject, the replacement of the background makesit possible to create a natural background replaced image that adifferent background appears through a part of the subject.

Further, according to the image taking system 1 of the presentembodiment, a revision is applied to a background replaced image inwhich a background is replaced by a desired background. This featuremakes it possible to create the background replaced image in which abackground is replaced more naturally.

In the above description, as an example of the image revision section inthe background replacing apparatus, there is raised and explained theimage revision section 760 that deals with the reduction amount on theinitial subject mask and the initial background mask as the revisionamount to the background replaced image. However, the present inventionis not restricted to this embodiment. It is acceptable that the imagerevision section of the present invention is ones as set forth below.

FIG. 38 is an explanatory view useful for understanding details ofanother image revision section different from the image revision sectionshown in FIG. 37.

Hereinafter, there will be described different points from the imagerevision section 760 shown in FIG. 37. In FIG. 38, the same referencenumbers as those of FIG. 37 denotes the same parts, and the redundantdescription will be omitted. In the following description, the G-valuehistogram H3 shown in the part (c) of FIG. 17 will be referred to.

An image revision section 810 shown in FIG. 38 has a revision valueinput section 811 as set forth below, different from the image revisionsection 760 shown in FIG. 38.

The initial subject mask and the initial background mask are set up, asmentioned above, in accordance with two thresholds Sr1 and Sr2, whichare computed from the G-value histogram H3 created from the correctedback light photographic image. More in details, set up as the initialsubject mask is an area consisting of pixels in which G-value offers thefirst threshold Sr1 or less, of a plurality of pixels constituting thecorrected back light photographic image, and set up as the initialbackground mask is an area consisting of pixels in which G-value offersthe second threshold Sr2 or more, of a plurality of pixels constitutingthe corrected back light photographic image.

According to the image revision section 810 shown in FIG. 38, treated asthe revision values are new two thresholds instead of the two thresholdsSr1 and Sr2, which are computed from the G-value histogram H3, set up asthe revision value for the initial subject mask is the threshold that isthe first threshold Sr1 or less, and set up as the revision value forthe initial background mask is the threshold that is the secondthreshold Sr2 or more. Two thresholds, which are treated as the revisionvalues in the image revision section 810, correspond to an example ofthe “distinction reference” referred to in the present invention.

The revision value input section 811 causes the display screen 220 a todisplay an input image plane. On the input image plane, those twothresholds are entered through an operator's operation of the keyboard230 and the mouse 240. The revision value input section 811 transfersthe entered two thresholds to the subject mask set up section 731 c andthe background mask set up section 731 d, which are also shown in FIG.16. Thus, there are set up new subject mask and background mask, whichare expanded in a boundary area as compared with those concerned withthe subject mask and the background mask set up using the firstthreshold Sr1 and the second threshold Sr2. As a result, the final twomasks are also newly formed, so that the portion that the backgroundappears through a part of the subject is much included in the boundaryarea as compared with a case where the original two initial masks areused.

In effect, the use of the image revision section 810 shown in FIG. 38also makes it possible to obtain the same affect as the image revisionsection 760 that deals with the reduction amount.

According to the present embodiment, the retrieval range, which is setup when the subject color and the combination ratio α on a certain pixelin the boundary area are presumed, is a range on the pixel, and is arange that includes pixels of an area defined by the subject mask M1′,the number of which pixels exceeds a predetermined number. Hence, theretrieval range is extended as a position of a pixel goes away from thearea defined by the subject mask M1′. Hereinafter, there will beexplained a method of setting up the retrieval range suppressing theexpansion of the retrieval range as mentioned above, which is differentfrom the present embodiment.

First, according to the different set up method, the presumption of thesubject color and the combination ratio α is carried out from pixelscloser to the area defined by the subject mask M1′ in the boundary area.

FIG. 39 is a typical illustration useful understanding a state that thepresumption of the subject color and the combination ratio α is carriedout from pixels closer to the area defined by the subject mask M1′ inthe boundary area.

As shown in FIG. 39, according to the different set up method, thepresumption of the subject color and the combination ratio α is carriedout from the side of the outline L3 of the subject mask M1′. Accordingto the different set up method, the pixels, which are presumed in thesubject color, are regarded as the pixels in the area defined by thesubject mask M1′. As a result, in the boundary area, there are formedapparent outlines (dotted lines L3′) which are equal to the outline L3of the subject mask M1′. The apparent outlines advance to the boundaryarea with the progress of the presumptions, as shown in FIG. 39.

According to the different set up method, setting up of the retrievalrange is carried out using the apparent outlines (dotted lines L3′)advanced to the boundary area.

FIG. 40 is a typical illustration useful understanding a state thatsetting up of the retrieval range is carried out using the apparentoutlines.

As shown in FIG. 40, according to the different set up method, as aretrieval range T2″, there is set up a range that includes pixels of anarea of the subject side starting from the apparent outline (dotted lineL3′), the number of which pixels exceeds a predetermined number in thearea of the subject side, on the pixel G in which the subject color andthe combination ratio α are presumed. In an area B2′ of the subject sidestarting from the apparent outline, in the retrieval range T2″, thepresumption of the subject color is carried out. In the presumption, thesubject color, which is already presumed on pixels of the subject sidestarting from the apparent outline, is used as a candidate of thepresumption.

In this manner, utilization for the presumption of the area B2′ of thesubject side starting from the apparent outline progressing in theboundary area whenever the presumption advances makes it possible tosuppress the enlargement of the retrieval range on pixels of thepositions apart from the outline L3 of the subject mask M1′ too.

Incidentally, in the above description, as an embodiment of thebackground replacing apparatus according to the present invention, thereis shown, by way of the example, the background replacing apparatus 700in which created are two masks of the subject mask defining a rangewherein the subject occupies in the corrected back light photographicimage and the background mask defining a range wherein the backgroundoccupies in the corrected back light photographic image, and those twomasks are used to perform a replacement of the background in thecorrected follow light photographic image. However, the presentinvention is not restricted to this embodiment. As will be describedhereinafter, for example, it is acceptable that a background replacingapparatus of the present invention creates only the background mask anduses only the background mask to perform a replacement of the backgroundin the corrected follow light photographic image.

Hereinafter, there will be explained an example of a backgroundreplacing method using only the background mask. It is noted that thebackground mask is created in the same manner as a creation of thebackground mask by the mask creating section 731 shown in FIG. 15, andthus the redundant explanation for the creation of the background maskwill be omitted.

In the background replacement using only the background mask, first,there is created a non-background mask, which defines an area exceptingan area defined by the background mask on the corrected follow lightphotographic image, in accordance with the background mask. The areadefined by the non-background mask is a combination area of an areadefined by the subject mask and the boundary mask. On all the pixels inthe area defined by the non-background mask, there is performed apresumption of the background color, the subject color and thecombination ratio of those colors. It is noted that the background maskis presumed in the same manner as a presumption of the background maskby the background color presuming section 732 a shown in FIG. 27, andthus the redundant explanation for the presumption of the backgroundcolor will be omitted. Hereinafter, there will be explained mainly amethod of presuming the subject.

FIG. 41 is an illustration useful for understanding a state that when asubject color is estimated on a certain pixel in an area defined by anon-background mask, of the corrected follow light photographic image, asearch range is set up on the pixel. FIG. 41 shows, by way of theexample, a case where a combination area of the area defined by thesubject mask M1′ shown in FIG. 22 and the boundary area M3 is equivalentto an area defined by a non-background mask M4. The area defined by thenon-background mask M4 corresponds to an example of “other areasexcepting the background area in the photographic image” referred to inthe present invention.

In order to set up a retrieval range on a certain pixel G in the areadefined by the non-background mask M4, first, the retrieval range isgradually expanded starting from a square-like shaped predeterminedinitial range T3 on the pixel G. The expansion is stopped when thepixels in the area defined by the non-background mask M4 in thecorrected follow light photographic image exceeds a predetermined numberof pixels and are included in the retrieval range, and a retrieval rangeT3′ at that time is set up as a retrieval range for the subjectpresumption.

When the retrieval range T3′ is set up, the subsequent area in thecorrected follow light photographic image is confirmed as an area usedfor presuming the subject color on the pixel G.

FIG. 42 is an illustration showing an example of an area used forestimating a subject color on a certain pixel.

As seen from FIG. 42, as the area used for presuming the subject coloron the pixel G, there is confirmed an area B3 in the corrected followlight photographic image, which is the area defined by thenon-background area, and is in the retrieval range T3′ set up in themanner as mentioned above.

Thus, when the area B3 used for presuming the subject color on the pixelG is confirmed, first, the color of the pixel G is confirmed, and thenthe subject color on the pixel G is retrieved, from among colors of aplurality of pixels in the area B3, in accordance with a retrievalmethod as set forth below.

FIG. 43 is an illustration useful for understanding a method ofsearching a subject color on a certain pixel in an area defined by anon-background mask, of the corrected follow light photographic image.

FIG. 43 shows, as to a certain pixel in a boundary area of the correctedfollow light photographic image, a straight line CL3 coupling abackground color C7 presumed with a color C8 of the pixel on the RGBcolor space.

The area defined by the non-background mask in the corrected followlight photographic image includes the boundary area as mentioned above.There is a possibility that the boundary area includes a portion that abackground appears through a part of the subject at the time ofphotography, and as to the color of the pixel of such a portion, thebackground color is combined with the subject color into a combinationcolor. In the event that the color of the pixel to be retrieved in thesubject color is such a combination color, the color C8 of the pixel,which is the combination color, the background color C7 as the base ofthe color C8, and the subject color are lined up on the RGB color space.

Thus, there is first retrieved, from among colors of a plurality ofpixels in the area B3, a color C9 lined up on the extended line of thestraight line CL3. In the event that there exists a plurality of colorsC9, there is adopted one color from among those colors C9, as thesubject color.

The area defined by the non-background mask includes pixels havingcolors other than the subject colors, such as the pixel having thecombination color, and the pixel having only the background. Hence,there is a high possibility that of the plurality of colors C9, thecolor farthest from the color C8 of the pixel is the subject color thatis the base of the color C8 of the pixel together with the backgroundcolor C7.

Thus, according to a method of retrieving the subject color from amongthe colors of the pixels in the area defined by the non-background mask,in the event that a plurality of colors C9 is lined up on the extendedline of the straight line CL3, there is adopted the color farthest fromthe color C7 of the pixel, from among the plurality of colors C9, as asubject color C10 on the pixel.

When the subject color is presumed in the manner as mentioned above, asshown in FIG. 43, the ration of length of a line segment coupling on theRGB color space the background color C7 presumed by the background colorpresuming section 732 a with the subject color C10 presumed by thesubject color presuming section 732 b to length of a line segmentcoupling the background color C7 with the color C8 of the pixel iscomputed in form of a combination ratio α where the color C8 of thepixel is combined with the background color C7 and the subject colorC10.

According to the background replacement using only the background mask,processing from the set up of the retrieval range as mentioned above upto the computation of the combination ratio is executed on all thepixels in the area defined by the non-background mask in the correctedfollow light photographic image.

When the presumption of the background color, the subject color and thecombination ratio, as mentioned above, is completed, first, the areadefined by the background mask in the corrected follow lightphotographic image is replaced by a different background of a customer'swish. Next, as to all the pixels in the area defined by thenon-background mask in the corrected follow light photographic image,the combination color is determined in accordance with the methodexplained with reference to FIG. 31. Colors of all the pixels in thearea defined by the non-background mask are replaced by the combinationcolor to complete the background replaced image.

As to the pixels of the area occupied by only the subject in the areadefined by the non-background mask, the combination ratio α issubstantially “1”. As a result, when the background is replaced, thecolor of the subject per se in the pixel is maintained. As to the pixelsof the area occupied by only the background in the area defined by thenon-background mask, the combination ratio α is substantially “0”. As aresult, when the background is replaced, the color of the pixel iscompletely replaced by a different background color.

Also the background replacement using only the background mask, which isexplained with reference to FIG. 41 to FIG. 43, makes it possible, in asimilar fashion to the image taking system 1, which is explained withreference to FIG. 1 to FIG. 36, to create a background replaced imagewherein the background in the photographic image is naturally replacedby a background of a client wish.

For example, according to the background replacement using both thesubject mask and the background mask, in the event that there is comingout of light emission color of the EL panel 130 in the area defined bythe subject mask, there is a possibility, at the time of the backgroundreplacement, that coming out of light emission color of the EL panel 130remains. According to the background replacement using only thebackground mask, however, it is possible to exactly estimate the subjectcolor removing a color of coming out, since all the areas other than thearea of the complete background color is the presumption object.

The above description is concerned with the embodiment in which thebackground replacement is carried out in accordance with either one ofmethods of the background replacement using both the subject mask andthe background mask, and the background replacement using only thebackground mask. It is noted, however, that the present invention is notrestricted to the present embodiment.

According to the background replacement using both the subject mask andthe background mask, and the background replacement using only thebackground mask, it is possible to estimate the subject color insubstantially similar fashion. However, the area B2, which is shown inFIG. 33 used for presumption of the subject color in the former method,is narrower in the range than the area B3, which is shown in FIG. 40used for presumption of the subject color in the later method.Therefore, in some situation of the photographic image, according to theformer method, it happens that there is detected no candidate of thesubject color such as the colors C3 arranged on the extended line of thestraight line CL1 shown in FIG. 34.

Accordingly, it is considered such a presumption method that first, thebackground replacement using both the subject mask and the backgroundmask is adopted to perform the presumption of the subject color. In theevent that there is detected no candidate of the subject color, thebackground replacement using only the background mask is adopted toperform the presumption of the subject color, so that the candidate ofthe subject color is detected from the area B3 shown in FIG. 40, whichis larger in range than the area B2 shown in FIG. 33.

The accuracy of the presumption of the subject color according to theabove-mentioned two presumption methods is substantially same as oneanother. In some situation of the photographic image, however, ithappens that either one of the above-mentioned two presumption methodsoffers greater accuracy somewhat. Consequently, for example, it isconsidered that the above-mentioned two types of presumption methods arecombined in the manner as set forth below to emphasize the presumptionaccuracy.

For example, in the event that the subject color and the combinationratio α are estimated in accordance with the above-mentioned twopresumption methods, or in the event that only the combination ratio α,which is obtained in accordance with either one of the above-mentionedtwo presumption methods, is closer to “0” and “1”, for some reason,there is a high possibility that accuracy of the presumption accordingto the presumption method is lower than accuracy of the presumptionaccording to another presumption method. Accordingly, there isconsidered such a method that the subject color and the combinationratio α are estimated in accordance with the above-mentioned twopresumption methods, and it is determined whether only one combinationratio α of two combination ratios α offers a predetermined firstthreshold close to “0” or a predetermined second threshold close to “1”,and in the event that it is decided that only one combination ratio α oftwo combination ratios α offers the predetermined second threshold closeto “1”, the subject color associated with another combination ratio α isadopted.

Further, in order to obtain the presumption result with the averageprecision between the above-mentioned two presumption methods, there isconsidered a method in which the average color of two sorts of subjectcolors, which are presumed in accordance with the above-mentioned twopresumption methods, is adopted as the subject colors.

According to the above-mentioned three ways of presumption methods, itis possible to emphasize the presumption accuracy as compared with acase where the subject color is estimated by either one of thebackground replacement using both the subject mask and the backgroundmask and the background replacement using only the background mask.

Incidentally, the above-mentioned explanation is concerned with anexample assuming that a background to be replaced by a customer's wishis one. It is acceptable, however, that a background replacing apparatusof the present invention is constructed, for example, in such a mannerthat the number of backgrounds to be replaced by a customer's wish is aplural and the background replacing apparatus is capable of effectivelycoping with the plurality of backgrounds to be replaced, as will beexplained hereinafter.

FIG. 44 is a view of a replacement processing section that is capable ofeffectively coping with the plurality of backgrounds to be replaced,which is different from the replacement processing section 730 shown inFIG. 15. In FIG. 44, the same parts are denoted by the same referencenumbers as those of FIG. 15, and the redundant explanation will beomitted.

A replacement processing section 730′ shown in FIG. 44 is constructed insuch a manner that the replacement processing section 730 shown in FIG.15 additionally includes a storage section 734 for storing the presumedsubject color and the combination ratio α. The storage section 734corresponds to an example of the storage section referred to in thepresent invention.

In the replacement processing section 730′, the mask creating section731 creates a background mask and a subject mask on a photographicimage, and the color presuming section 732 presumes the subject colorand the combination ratio α. The background mask, the subject mask, thesubject color, and the combination ratio α are stored in the storagesection 734. Further, in the replacement processing section 730′, abackground replacing image creating section 733′ creates a backgroundreplacing image using the background mask, the subject mask, the subjectcolor, and the combination ratio α stored in the storage section 734.

According to the replacement processing section 730′ shown in FIG. 44,when once the background mask, the subject mask, the subject color, andthe combination ratio α are determined, even if a plurality ofbackgrounds to be replaced is concerned, the background mask, thesubject mask, the subject color, and the combination ratio α are appliedto all the backgrounds to be replaced. Thus, it is possible, forexample, to save trouble for determining the background mask, thesubject mask, the subject color, and the combination ratio α, andthereby effectively creating the background replacing image.

Further according to the embodiment as mentioned above, as shown in FIG.6, there is explained the color correcting section 724 for applying thecolor correcting processing to color of the effective image that iscreated in accordance with the follow light photographic image, of theimage correcting section 720. However, the present invention is notrestricted to the present embodiment. According to a backgroundreplacing apparatus of the present invention, it is acceptable that thereplacement processing section applies the color correcting processingto the presumed subject color, but not to the color of the effectiveimage, as will be explained hereinafter.

FIG. 45 is a view of a replacement processing section having a colorcorrecting section for applying the color correcting processing to thepresumed subject color, which is different from the replacementprocessing section 730 shown in FIG. 15. In FIG. 45, the same parts aredenoted by the same reference numbers as those of FIG. 15, and theredundant explanation will be omitted.

A replacement processing section 730″ shown in FIG. 45 is constructed insuch a manner that the replacement processing section 730 shown in FIG.15 additionally includes a parameter determining section 735 fordetermining a correction parameter used in color correction for thesubject color, and a color correction section 736 for applying the colorcorrection to the subject color, using the correction parameterdetermined by the parameter determining section 735. The parameterdetermining section 735 shown in FIG. 45 corresponds to an example ofthe first parameter determining section, the second parameterdetermining section, the third parameter determining section, and thefourth parameter determining section referred to in the presentinvention. The color correction section 736 shown in FIG. 45 correspondsto an example of the first color correction section, the second colorcorrection section, the third color correction section, and the fourthcolor correction section referred to in the present invention.

In the replacement processing section 730″, the color correction section736 performs a color correction addressed as a so-called set up in whichthe color shade and the white balance are corrected for a color of thearea defined by the subject area and the presumed subject color to thecolor shade and the white balance which are generally deemed to bepreferable for a person. The parameter determining section 735determines a correction parameter (a set up parameter) which is used forthe set up.

The parameter determining section 735 shown in FIG. 45 has a function ofdetermining the set up parameter in accordance with the effective imagecreated on the basis of the follow light photographic image in theeffective image creating section 723, a function of determining the setup parameter in accordance with the portion other than the background ofthe image in which the color of the boundary area is replaced with thesubject color presumed in the color presuming section 732, a function ofdetermining the set up parameter to a parameter according to anoperation, and a function of determining the set up parameter inaccordance with a background for replacement. It is determined by anoperator's operation through an operational screen (not illustrated) asto what function of the above-mentioned four types of functions is usedto determine the set up parameter. The set up using the set upparameter, which is determined by the parameter determining section 735,is applied to the color of the area defined by the subject area and thepresumed subject color, as mentioned above.

For example, in the photo-studio 10 shown in FIG. 1, it happens that insome situation of photography, the white balance at the time ofphotography by the digital camera 11 may be affected by the indigo bluewhich is the luminescent color of the EL panel 130. In this case, forexample, a person's face may come out paler than life. However,according to the set up by the color correction section 736 shown inFIG. 45, it is possible to correct the color of the person's face to apreferable color.

As in the color correcting section 724 shown in FIG. 6, in the eventthat the color correcting processing is applied to the effective imagethat is created in accordance with the follow light photographic image,in some situation of the photographic image, it may happen that thepresumption of the subject color is carried out incorrectly, forexample, by the reason that the gradation of the photographic image isaltered. Further, in the event that the set up is applied to the imageafter the background replacement, there is a possibility that a color ofthe background for replacement is also altered so that an image ofbackground, which is different in color shade from a customer's wishedbackground. According to the set up by the parameter determining section735 and the color correction section 736, which are shown in FIG. 45,the set up is carried out after the presumption of the subject color andbefore the replacement of the background. Thus, it is possible toperform the set up without having an effect on the presumption of thesubject color and the color of the background for the replacement, andthereby obtaining a high picture quality of image in which a backgroundis replaced.

Further, according to the example shown in FIG. 45, applied to thebackground for the replacement, which is stored in the server 40, isrelation information representative of a relative positional relation inan image between the subject and the background and a relative size ofthe subject to the background in the image. When a background replacedimage creating section 733″ of FIG. 45 creates a background replacedimage using a background for replacement, which is a client's wishedbackground, the background replaced image creating section 733″ createsthe background replaced image in such a manner that the relativerelation in the background replaced image between the subject and thebackground is the same relative relation as the relative relation whichis applied to a background for replacement, which is a client's wishedbackground. This processing makes it possible to obtain a backgroundreplaced image enhanced in naturalness to look at in which for example,position and size of the subject in the background replaced image arenatural ones to the background.

Next, there will be explained a second embodiment.

The second embodiment is different from the first embodiment in pointssuch as the structure of the photo-studio of the image taking system,the photographic processing to be carried out in the photo-studio andthe background replacement processing to be carried out in the personalcomputer. Hereinafter, there will be described different points from thefirst embodiment. In following figures, the same reference numbers asthose of FIG. 1 to FIG. 37 denote the same parts, and the redundantdescription will be omitted.

FIG. 46 is a construction view of an image taking system, to which asecond embodiment of a background replacing apparatus of the presentinvention is applied.

An image taking system 1001 shown in FIG. 38 comprises: a photo-studio1010 for obtaining a photographic image through photography of thesubject; a personal computer 20 that is operated in form of a secondembodiment of a background replacing apparatus of the present invention,in which the subject of the photographic image is distinguished from thebackground, so that the background of the photographic image is replacedwith a desired background to create a background replaced image; aprinter 30; and a server 40. The photo-studio 1010 comprises a digitalcamera 1011, a flashlight emitting device 12, an EL(Electro-Luminescence) panel 1130, and a power source 14 for driving theEL panel.

The digital camera 1011 has a multi-page function, in which when anoperator depresses a shutter button once, three times photography for asubject P is automatically carried out for a short time. A photographicimage, which is obtained through a first photography, is temporarilystored in a memory of the digital camera 1011. In the image takingsystem 1001, the digital camera 1011 is connected to the personalcomputer 20, and at the time point that a third photography iscompleted, a photographic image, which is obtained through the thirdphotography, is directly transmitted to the personal computer 20.

The digital camera 1011 is provided with a panel turn-on control circuit11 a and a flashlight control circuit 11 b, which will be describedlater. The digital camera 1011 is connected via the panel turn-oncontrol circuit 11 a to the power source 14, and is also connected viathe flashlight control circuit 11 b to the flashlight emitting device12. In the multi-page photography, the digital camera 1011 transmits,for every photography, first, to the power source 14 a panel turn-oninstruction signal that instructs the power source 14 for the EL panel1130 to turn on the EL panel 1130, and then transmits to the flashlightemitting device 12 a flashlight instruction signal that instructs theflashlight emitting device 12 to emit the flashlight. The panel turn-oncontrol circuit 11 a blocks the panel turn-on instruction signal, whichis emitted from the digital camera 1011 for every time of a multi-pagephotography, at the time of a first photography, and permits the panelturn-on instruction signal to pass at the time of second and thirdphotography. The flashlight control circuit 11 b permits the flashlightinstruction signal to pass at the time of first and second photography,and blocks the flashlight instruction signal at the time of the thirdphotography. As a result, as to the multi-page photography: at the timeof the first photography, there is performed only input of theflashlight instruction signal to the flashlight emitting device 12; atthe time of the second photography, first, the panel turn-on instructionsignal is inputted to the power source 14, and then the flashlightinstruction signal is inputted to the flashlight emitting device 12; andat the time of the third photography, there is performed only input ofthe panel turn-on instruction signal to the power source 14.

Upon receipt of the flashlight instruction signal from the digitalcamera 1011, the flashlight emitting device 12 emits flashlight to thesubject P. As mentioned above, the flashlight emitting device 12receives the flashlight instruction signal only at the time of the firstand second photography in the multi-page photography in the digitalcamera 1011. Thus, the flashlight emitting device 12 emits theflashlight at the time of the first and second photography only, and theflashlight emitting device 12 turns off at the time of the thirdphotography.

The EL panel 1130 comprises: a cover member 131 having a loading plane131 a for loading with the subject P and a rear plane 131 b located atthe back of the subject P, which are transparent; and a dispersion typeof EL device 1132 that is housed in the cover member 131. A power source14 serves as a power source for applying a driving voltage to thedispersion type of EL device 1132. Further as mentioned above, the powersource 14 receives the panel turn-on instruction signal only at the timeof the second and third photography in the multi-page photography in thedigital camera 1011. Thus, the power source 14 turns on the EL panel1130 at the time of the second photography. When the power source 14receives the panel turn-on instruction signal at the time of the thirdphotography, the power source 14 maintains a turn-on of the EL panel1130.

According to the general dispersion type of EL device, it is possible toadjust the luminescent color to various colors including white throughfurther mixing of a plurality of fluorescent powders mutually differentfrom one another in luminescent color at the time of the manufacture. Tothe contrary, according to the dispersion type of EL device 1132 of thepresent embodiment, the luminescent color is adjusted by the fluorescentpowders of red. As a result, the color of the EL panel 1130 offers thinred at the time of the turn-off, and offers, at the time of lightemission, blue that is adjusted in indigo blue which is the luminescentcolor of the dispersion type of EL device 1132.

According to the photo-studio 1010 of the present embodiment,photography by the digital camera 1011 is carried out, as mentionedabove, in such a way that the first photography is carried out in thestate that the EL panel 1130 turns off and the flashlight emittingdevice 12 emits flashlight, the second photography is carried out in thestate that the EL panel 1130 turns on and the flashlight emitting device12 emits flashlight, and the third photography is carried out in thestate that the EL panel 1130 turns on and the flashlight emitting device12 turns off. Further, according to the photo-studio 1010 of the presentembodiment, the luminance of the flashlight emitted from the flashlightemitting device 12 is extremely higher as compared with luminance whenthe EL panel 1130 turns on. As a result, according to the secondphotography as well as the first photography, there is obtained a followlight photographic image, in which the subject P is clearlyphotographed, through photography in a follow light state that aflashlight, which is higher in luminance than the luminance of the ELpanel 1130, is projected from the side of the digital camera 1011 to thesubject P. Further, as mentioned above, the color of the EL panel 1130offers thin red at the time of the turn-off, and offers blue at the timeof light emission. As a result, as to the follow light photographicimage associated with the first photography, the background offers thinred, and as to the follow light photographic image associated with thesecond photography, the background offers blue. Two follow lightphotographic images, which are mutually different in the backgroundcolor, correspond to an example of the “plurality of photographic imagesgenerated through photography of a common subject under a plurality ofphotographic conditions mutually different in color of background”.According to the third photography, there is obtained a back lightphotographic image, in which an image of the subject P is biased to theshadow side and an image of the EL panel 1130 is biased to the highlightside, through photography in a back light state that only light from theEL panel 1130 is projected from the back to the subject P. In thismanner, according to the present embodiment, the multi-page photographyof the digital camera 1011 makes it possible to obtain threephotographic images of two follow light photographic images, which aremutually different in the background color, and one back lightphotographic image.

Next, there will be explained a flow of work in which a secondembodiment of a background replacing method of the present invention,which is carried out by the image taking system 1001 shown in FIG. 46,is applied. In the following figures, the same reference numbers asthose of FIG. 46 denotes the same parts.

FIG. 47 is a flowchart useful for understanding a second embodiment of abackground replacing method of the present invention, which is carriedout by the image taking system shown in FIG. 46.

The work shown in the flowchart of FIG. 47 is carried out on theassumption that in the photo-studio 1010, the subject P, the digitalcamera 1011 and the flashlight emitting device 12 are arranged at theirsuitable positions, respectively, and the EL panel 1130 turns off.

When an operator depresses a shutter button of the digital camera 1011after adjustments of focus and exposure, the photography for the subjectP is carried out continuously three times.

First, in the first photography, the flashlight emitting device 12 is inresponse to the flash instruction signal transmitted from the digitalcamera 1011 to emit flashlight, so that photography for the subject P iscarried out in the state of the follow light by the flashlight (stepS201). At the time of the first photography, the EL panel 1130 turns offto offer the thin red.

In the second photography, first, the EL panel 1130 is in response tothe panel turn-on instruction signal transmitted from the digital camera1011 to turn on (step S202). Next, the flashlight emitting device 12 isin response to the flash instruction signal transmitted from the digitalcamera 1011 to emit flashlight, so that photography for the subject P iscarried out in the state of the follow light by the flashlight (stepS203). At the time of the second photography, the EL panel 1130 emitslight with blue.

Subsequently, the third photography is carried out in a state that theflashlight emitting device 12 turns off, that is, in the back lightstate by the illumination of only the EL panel 1130 (step S204).

Three photographic images of two follow light photographic imagesmutually different in the background color and the back lightphotographic image, which are obtained through the step S202 to the stepS204, respectively, are temporarily stored in a memory of the digitalcamera 1011.

Next, there will be explained processing following step S205 of theflowchart of FIG. 39. Of the processing, which will be describedhereinafter, processing of the step S205 to the step S207 corresponds tothe second embodiment of a background replacing method of the presentinvention.

When the third photography is carried out through the processing of thestep S204 and the storage of the photographic image obtained through thethird photography into the memory of the digital camera 1011 iscompleted, three photographic images obtained through three timesphotography, which are temporarily stored in the memory, are immediatelytransferred to the personal computer 20. In step S205, when a customeroperates the personal computer 20, a desired background is read out fromthe server 40 or some input storage media. The obtaining processing forthe photographic images and the backgrounds, which are executed in thestep S205, corresponds to an example of the image obtaining step in thebackground replacing method of the present invention.

In step S206, the three photographic images are subjected to the imagecorrection processing including the correction of the trapezoidaldistortion.

In step S207, there is carried out a background replacing processing (itwill be described later), in which a background replaced image iscreated in accordance with the corrected photographic images. Here, inthe background replacing processing to be carried out in the step S207,examples of an area distinguishing process, a mixing state presumingprocess and a background replacing process in a background replacingmethod of the present invention are executed, respectively.

Details of the respective processing of the second embodiment of abackground replacing method of the present invention from the step S205to the step S207 will be described in conjunction with functions of therespective section of the second embodiment of a background replacingmethod of the present invention.

Next, image data representative of the background replaced image createdin the background replacing processing (the step S207) is firsttransferred to the image display unit 220, and then transferred to atleast one of the printer 30 and the output storage media of a customer'swish (step S208). In the image display unit 220, the background replacedimage is displayed on the display screen 220 a in accordance with thetransferred image data (step S209). In the event that the image data istransferred to the printer 30, the printer 30 prints the backgroundreplaced image in accordance with the image data (step S210).

The processing of the step S205 to the step S209 including the secondembodiment of the background replacing method of the present inventionis executed in the personal computer 20.

Next, there will be described a second embodiment of a backgroundreplacing program related to the present invention, and a secondembodiment of a background replacing program storage medium of thepresent invention.

FIG. 48 is a view useful for understanding a second embodiment of abackground replacing program storage medium of the present inventionstoring a second embodiment of a background replacing program related tothe present invention.

A CD-ROM 1510 shown in FIG. 48 stores therein a background replacingprogram 600, and corresponds to a second embodiment of a backgroundreplacing program storage medium of the present invention.

The background replacing program 600 comprises an image obtainingsection 610, an image correcting section 620, a replacement processingsection 1630, an image data transmission section 640, and an imagedisplay section 650.

As to details of every element of the background replacing program 1600,they will be described in conjunction with functions of respectivesections of the background replacing apparatus of the present invention.In the following figures, the same reference numbers as those of FIG. 46and FIG. 48 denote the same parts.

FIG. 49 is a functional block diagram of elements for operating thepersonal computer shown in FIG. 46 and FIG. 3 in form of a secondembodiment of a background replacing apparatus of the present invention,when the background replacing program shown in FIG. 48 is installed inthe personal computer shown in FIG. 46 and FIG. 3. FIG. 49 also shows:the photo-studio 1010 comprising the digital camera 1011, the flashlightemitting device 12, and the EL panel 1130; the printer 30; and theserver 40.

According to a background replacing apparatus 1700 shown in FIG. 49,there is executed such background replacing processing that the digitalcamera 1011 of the photo-studio 1010 is used to obtain two follow lightphotographic images mutually different in the background color, whichare photographed under the follow light, and a back light photographicimage, which is photographed under the back light, and a subject and abackground in a predetermined one of the two follow light photographicimages are distinguished from one another in accordance with the backlight photographic image, of those three photographic images, so thatthe background is replaced by a desired background. The backgroundreplacing apparatus 1700 comprises an image obtaining section 710, animage correcting section 720, a replacement processing section 1730, animage data transmission section 740, and an image display section 750.

Hereinafter, there will be explained the present embodiment on theassumption that the background replacing processing is carried out onthe follow light photographic image that is thin red in the backgroundcolor, that is, the follow light photographic image obtained through thefirst photography in the multi-page photography, of the two follow lightphotographic image as mentioned above. However, the present invention isnot restricted to the present embodiment, and it is acceptable that thebackground replacing processing is carried out on the follow lightphotographic image that is blue in the background color, that is, thefollow light photographic image obtained through the second photographyin the multi-page photography.

The replacement processing section 1730 performs a backgroundreplacement processing in such a manner that a background replaced imageis created in accordance with the three photographic images subjected tothe image correcting processing by the image correcting section 720. Thereplacement processing section 1730 is substantially constructed in sucha manner that the CPU 211 of the personal computer 20 operates inaccordance with the replacement processing section 1630 of thebackground replacing program 1600. As to details of the replacementprocessing section 1730, it will be described later in conjunction withdetails of the replacement processing with reference to other figures.The replacement processing section 1730 corresponds to an example of oneserving as the area discriminating section, the mixing state presumingsection and the background replacing section of the background replacingapparatus of the present invention.

Next, hereinafter, there will be described details of the replacementprocessing section 1730 shown in FIG. 49.

The replacement processing section 1730 replaces the background of thecorrected follow light photographic image associated with the firstphotography, of the two corrected follow light photographic imagesmutually different in the background color. At that time, thereplacement processing section 1730 first discriminates each of the twocorrected follow light photographic images among a subject area, abackground area, and a boundary area interposed between the subject areaand the background area. The replacement processing section 1730 retainsthe subject area of the corrected follow light photographic imageassociated with the first photography, and replaces the background areaby another background of a customer's wish.

The replacement processing section 730 presumes, on the assumption thatthe whole pixels of the boundary area of the corrected follow lightphotographic image have the combination color as mentioned above,through using the two corrected follow light photographic images, thesubject color (corresponding to an example of the subject color referredto in the present invention) that is the base of a combination of colorsof the pixels, which is common between the two corrected follow lightphotographic images, and the combination ratio (corresponding to anexample of the mixing ratio referred to in the present invention) of thesubject color and the respective background color.

Thus, the replacement processing section 1730 replaces, as to thecorrected follow light photographic image associated with the firstphotography, the colors of the pixels of the boundary area by acombination color in which the presumed subject color is combined withthe different background color of the customer's wish at the presumedcombination ratio.

FIG. 50 is an explanatory view useful for understanding details of thereplacement processing section shown in FIG. 49. In the followingfigures, the same reference numbers as those of FIG. 49 denotes the sameparts.

The replacement processing section 1730 executes the above-mentionedbackground replacement processing. The replacement processing section1730 comprises a mask creating section 731, a color presuming section1732, and a background replaced image creating section 733. The maskcreating section 731, the color presuming section 1732, and thebackground replaced image creating section 733 correspond to theexamples of the area discriminating section, the mixing state presumingsection, and the background replacing section, of the backgroundreplacing apparatus of the present invention, respectively. The maskcreating section 731 executes an example of the area discriminating stepof the background replacing method of the present invention. The colorpresuming section 1732 executes an example of the mixing state presumingstep of the background replacing method of the present invention. Thebackground replaced image creating section 733 executes an example ofthe background replacing step of the background replacing method of thepresent invention.

First, there will be explained outlines of those structural elements.

The mask creating section 731 creates, as to the corrected back lightphotographic image, a subject mask that defines a range in which onlypixels of the subject color exist, and a background mask that defines arange in which only pixels of the background color exist. Those twomasks are created with a gap in such a way that the pixels of theportion that the background appears through the subject, as shown inFIG. 14 for instance, are excluded from the ranges defined by thosemasks.

The color presuming section 1732 applies the subject mask and thebackground mask, which are created by the mask creating section 731, tothe two corrected follow light photographic image which are mutuallydifferent in the background, and first, in the follow light photographicimage, recognizes the gap provided between those two masks, that is, theboundary area. The color presuming section 1732 presumes, on theassumption that the colors of the whole pixels in the boundary area arethe mixed color of the subject color and background color, and themixing state is the same under the different background color, throughusing the two corrected follow light photographic images, the subjectcolor that constitutes colors of the pixels, which is common between thetwo corrected follow light photographic images, and the combinationratio of the subject color and the background color.

The background replaced image creating section 733 retains, first, as tothe corrected follow light photographic image associated with the firstphotography, the area defined by the subject mask, and replaces the areadefined by the background mask with another background of a customer'swish. Next, the background replaced image creating section 733determines, as to each pixel of the boundary area, a combination colorthrough a combination of the color of the background of a customer'swish and the subject color presumed in the color presuming section 1732at the combination ratio presumed in the color presuming section 1732,and replaces the color of the respective pixel. This feature makes itpossible to create a background replaced image in which the backgroundof the corrected follow light photographic image is naturally replacedwith another background of a customer's wish.

The image data transmission section 740 of FIG. 49 transmits image datarepresentative of the background replaced image thus created to outputdevices and the output storage media 550 of FIG. 49.

In a similar fashion to that of the first embodiment, according to thesecond embodiment too, the mask creating section 731 creates the subjectmask M1′ and the background mask M2′ in accordance with a difference ofluminous intensity between the background and the subject. This featuremakes it possible to distinguish among the subject area, the backgroundarea and the boundary area in the photographic image. However, the areadiscriminating section referred to in the present invention is notrestricted to the present embodiment. It is acceptable that the areadiscriminating section referred to in the present invention is one asset forth below.

Hereinafter, there will be explained another embodiment of the areadiscriminating section referred to in the present invention.

According to the another embodiment of the area discriminating section,the subject area, the background area and the boundary area in thephotographic image are distinguished from one another in accordance withthe amount of the change in color among a plurality of photographicimages. The structure of the another embodiment of the areadiscriminating section is basically identical to that of the maskcreating section 731 shown in FIG. 16. According to the histogramcreating section 731 a of FIG. 16, there is created the histogram as toluminous intensity of the pixels constituting the corrected back lightphotographic image. To the contrary, according to this anotherembodiment of the area discriminating section, there is created, inaccordance with the above-mentioned two corrected follow lightphotographic images, the following histogram as to the amount of thechange in color between those two photographic images.

First, in order to create the histogram as to the amount of the changein color, according to this another embodiment of the areadiscriminating section, assuming that one of the two corrected followlight photographic images is denoted as a first photographic image, andanother is denoted as a second photographic image, the amount of thechange in color-phase of the pixels constituting the first photographicimage to the pixels of the associated positions in the secondphotographic image is adopted as a value representative of the amount ofthe change in color as mentioned above.

The histogram, which is created in accordance with this anotherembodiment, is a histogram in which the amount of the change incolor-phase of the pixels constituting the first photographic image isdivided into a large number of ranges, and the number of the pixelsdivided up onto each range is expressed by the frequency. The histogramof the change in color-phase is for example one wherein the horizontalaxis H1_1 of the luminance histogram H1 shown in the part (a) of FIG. 17is replaced by the amount of the change in color-phase. In the histogramof the change in color-phase, there appear two peaks on a portionwherein the amount of the change in color-phase is little and a portionwherein the amount of the change in color-phase is much. As mentionedabove, the difference between the two corrected follow lightphotographic images is only the background color. Thus, the peak on theportion, wherein the amount of the change in color-phase is much,corresponds to the background area of the first photographic image, andthe peak on the portion, wherein the amount of the change in color-phaseis little, corresponds to the subject area of the first photographicimage.

When the histogram of the change in color-phase is created as mentionedabove, according to this another embodiment of the area discriminatingsection, in accordance with the same procedure as that for the maskcreation by the mask creating section 731: the initial subject mask isset up in accordance with the peak on the portion wherein the amount ofthe change in color-phase is little; the initial background mask is setup in accordance with the peak on the portion wherein the amount of thechange in color-phase is much; and both the initial masks are reduced tofinally create the subject mask and the background mask, so that the useof those masks makes it possible to distinguish among the subject area,the background area and the boundary area, of the photographic image.

Thus, also according to the another embodiment of the areadiscriminating section as mentioned above, it is possible to distinguishamong the subject area, the background area and the boundary area, ofthe photographic image. According to the present embodiment, the use oftwo photographic images for a mask creation may involve somewhat anincrement of complication of processing. However, there is no need ofthe corrected back light photographic image, which is needed for a maskcreation by the mask creating section 731. In effect, according to thepresent embodiment, there is no need of the photographic processing ofthe step S204 in the flowchart of FIG. 39. As a result, the presentembodiment brings about such an advantage that a number of times forphotography can be reduced as compared with the example of theprocessing shown in the flowchart of FIG. 47.

The explanation for the present embodiment will be terminated, and nextthere will be explained details of the color presuming section 1732 ofthe replacement processing section 1730.

FIG. 51 is an explanatory view useful for understanding details of thecolor presuming section shown in FIG. 50.

The color presuming section 1732 comprises a background color presumingsection 732 a, a subject color presuming section 1732 b, and acombination ratio presuming section 1732 c.

With respect to the background color presuming section 732 a, it is thesame as the first embodiment, and thus the redundant explanation will beomitted.

The background color presuming section 732 a executes the presumption ofthe background color on each of two corrected follow light photographicimages, which are mutually different in the background color.

Next, there will be explained the subject color presuming section 1732 bshown in FIG. 51.

The subject color presuming section 1732 b estimates a subject colorthat is the origin of colors of pixels in a boundary area, which iscommon between two corrected follow light photographic images that aremutually different in a background color, using the background colorsdetermined by the background color presuming section 732 a on each ofthe pixels in the boundary areas of the two corrected follow lightphotographic images, as follows.

FIG. 52 is an illustration useful for understanding a method ofestimating a subject color that is the origin of colors of pixels in aboundary area, which is common between the corrected follow lightphotographic images that are mutually different in a background color.

FIG. 52 shows, as to a certain pixel in the boundary area of thecorrected follow light photographic image associated with the firstphotography, of the two corrected follow light photographic images, astraight line CL1 coupling between a first background color C1 presumedby the background color presuming section 732 a of FIG. 51, and a colorC2 (a first pixel color) of the pixel, on the RGB color space. Further,FIG. 52 shows, as to a pixel located at the position associated withsaid pixel in the corrected follow light photographic image associatedwith the second photography, a straight line CL2 coupling between asecond background color C3 presumed by the background color presumingsection 732 a of FIG. 51, and a color C4 (a second pixel color) of thepixel, on the RGB color space.

As mentioned above, there is a possibility that the color of pixels inthe boundary area is a combination color, in which at the time of thephotography, the background appears through a part of the subject, sothat the background color is combined to the subject color. In the eventthat a certain pixel color is such a combination color, the pixel color,the background color as the base of the pixel color, and the subjectcolor are lined up on the RGB color space. At that time, the twocorrected follow light photographic images are the common imagesexcepting a point that they are mutually different in the backgroundcolor, and thus the subject colors included in the pixels located at themutually associated positions between the two corrected follow lightphotographic images are the common color. Consequently, two straightlines CL1 and CL2 shown in FIG. 52, which are associated with the twopixels located at the mutually associated positions between the twocorrected follow light photographic images, respectively, intersect eachother at a common subject color C5 included in those pixels.

The subject color presuming section 1732 b shown in FIG. 51 determinesthe intersection of the two straight lines CL1 and CL2 as shown in FIG.52, as the subject color C5 on all the pixel pairs in the boundary area,of the pixel pairs each consisting of two pixels located at the mutuallyassociated positions between the two corrected follow light photographicimages. Consequently, according to the second embodiment, the subjectcolor C5 is uniquely determined, and there is no need of the retrievalas required for the first embodiment.

Here, there is terminated the explanation of the subject color presumingsection 1732 b shown in FIG. 51. Next, there will be explained thecombination ratio presuming section 1732 c shown in FIG. 51 inconjunction with FIG. 52.

The combination ratio presuming section 1732 c computes, as shown inFIG. 52, with respect to the corrected follow light photographic imageassociated with the first photography, the ratio of the length of a linesegment coupling on the RGB color space the first background color C1presumed by the background color presuming section 732 a with thesubject color C5 presumed by the subject color presuming section 1732 bto the length of a line segment coupling the first background color C1with the first pixel color C2 of the pixel, in form of a combinationratio α where the first pixel color C2 of the pixel is combined with thefirst background color C1 and the subject color C5. Likely, also withrespect to the corrected follow light photographic image associated withthe second photography, the combination ratio presuming section 1732 ccomputes the ratio of the length of a line segment coupling on the RGBcolor space the second background color C3 with the subject color C5 tothe length of a line segment coupling the second background color C3with the second pixel color C4 of the pixel, in form of a combinationratio α′ where the second pixel color C4 of the pixel is combined withthe second background color C3 and the subject color C5. In view of thefact that the first and second photography are carried out on amulti-page basis, it is considered that the states of mixing of thebackground color with the subject color in two corrected follow lightphotographic images are the same as one another, and thus the twocombination ratios α′ and α are substantially equal to one another.According to the present embodiment, as the photographic image, in whichthe background is directly replaced, there is adopted the correctedfollow light photographic image associated with the first photography.Consequently, the above-mentioned computation is carried out on only thecorrected follow light photographic image associated with the firstphotography.

The combination ratio presuming section 1732 c performs the computationof the combination ratio α on all the pixel pairs in the boundary areaof the two corrected follow light photographic images.

According to the second embodiment, as mentioned above, the use of thepresumed subject color and the combination color makes it possible tocreate the natural background replaced image in a similar fashion tothat of the first embodiment.

According to the embodiments of the present invention, as the example ofthe area discriminating section of the background replacing apparatus ofthe present invention, there is shown the mask creating section 731 thatcreates, as a histogram on luminous intensity of the pixels constitutingthe corrected back light photographic image, the histogram on theG-value, of the R-value, the G-value, and the B-value, which representcolors of the pixels. However, the present invention is not restrictedto the present embodiments. It is acceptable that the areadiscriminating section of the background replacing apparatus of thepresent invention creates the histogram on the R-value, it is acceptablethat the area discriminating section of the background replacingapparatus of the present invention creates the histogram on the B-value,or alternatively it is acceptable that the area discriminating sectionof the background replacing apparatus of the present invention createsthe histogram on the luminance of the pixels.

According to the embodiments of the present invention, as the example ofthe area discriminating section of the background replacing apparatus ofthe present invention, there is shown the mask creating section 731 thatcreates, a mask for distinguishing a background area of a photographicimage from other areas, in accordance with a single photographic imagereferred to a corrected back light photographic image. However, thepresent invention is not restricted to the present embodiments. It isacceptable that the area discriminating section of the backgroundreplacing apparatus of the present invention creates a mask fordistinguishing a background area of a photographic image from otherareas, in accordance with two photographic images that are mutuallydifferent in luminance and color of the background. In this case, thereis used a mask for distinguishing a portion wherein the amount of thechange in luminance and color is large relatively between the twophotographic images, from other areas. Thus, it is possible to create amask for distinguishing a background area of a photographic image fromother areas.

According to the embodiments of the present invention, as the example ofthe mixing ratio determining section of the background replacingapparatus of the present invention, there is shown the color presumingsection 732 in which as to a certain pixel in the boundary area, first,the subject color constituting the color of the pixel is presumed, andthen there is presumed, as the combination ratio, the ratio of mutualdistance among the background color, the color of the pixel, and thesubject color, in the RGB color space. However, the present invention isnot restricted to the present embodiment. It is acceptable that themixing ratio determining section of the background replacing apparatusof the present invention determines, for example, as to a certain pixelin the boundary area, a combination ratio in accordance with apositional relation between the pixel and the subject area, in otherwords, determines a combination ratio in such a way that the furtherpixel from the subject area is concerned, the ratio of the inclusion ofthe subject color is more decreased.

According to the embodiments of the present invention, as to the exampleof the image revision section of the background replacing apparatus ofthe present invention, the embodiments of present invention particularlyfail to refer to the input screen in which an operator inputs therevision value. It is acceptable that the input screen is such a type ofinput screen that an operator enters the numerical values through akeyboard. Alternatively, it is acceptable that the input screen is sucha type of input screen that there is provided a set up bar for revisionvalues, and an operator moves the set up bar through the mouse operationto enter the revision values.

According to the embodiments of the present invention, as the example ofthe image revision section of the background replacing apparatus of thepresent invention, there is shown the image revision section thatexecutes only the image revision by the revision for the subject areaand the background area. However, the present invention is notrestricted to the present embodiment. It is acceptable that the imagerevision section of the background replacing apparatus of the presentinvention performs the revision for colors of the respective portions ofthe created background replaced image as well as the revision for theabove-mentioned areas.

According to the embodiments of the present invention, as the example ofthe image revision section of the background replacing apparatus of thepresent invention, there is shown two image revision sections 760 and810 in which the subject mask and background mask are narrowed to revisethe background replaced image. However, the present invention is notrestricted to the present embodiment. It is acceptable that the imagerevision section of the background replacing apparatus of the presentinvention expands the subject mask and background mask, and/or narrowsthe boundary area, to revise the background replaced image. Thisrevision is effective to such an unnatural background replaced-imagethat the color of the background after the replacement is mixed up to aportion that the background is not seen originally. There is apossibility that such an unnatural background replaced-image occurs inthe event that the background after the replacement has strong colorssuch as a high chroma saturation of color.

According to the embodiments of the present invention, as the example ofthe image taking system in which a plurality of photographic images isphotographed, there is shown the image taking system 1 in which theflashlight emitting device 12 emits the flashlight at the time of thefirst photography, and the flashlight emitting device 12 turns off atthe time of the second photography, so that the follow light photographyand the back light photography are continuously performed in the namedorder. However, the present invention is not restricted to the presentembodiment. It is acceptable that the image taking system of the presentinvention is one in which the follow light photography and the backlight photography are continuously performed in reverse order to thenamed order. This type of image taking system can be constructed, forexample, in such a manner that in the embodiment as mentioned above, anintercept circuit, wherein an instruction signal that instructs theflashlight emitting device to emit flashlight, which is transmitted fromthe digital camera to the flashlight emitting device whenever the firstphotography is carried out, is intercepted at the time of the firstphotography, and passes at the time of the second photography, isinstalled on the transmission line for the instruction signal, whichconnects the digital camera with the flashlight emitting device.

According to the embodiments of the present invention, as the example ofthe image taking system in which a plurality of photographic images isphotographed, there is shown the image taking system 1 having theflashlight emitting device 12 in which when light emits once, it takes alot of time for charge in the subsequent light emission, and when thephotography is carried out continuously twice, the flashlight is emittedonly at the time of first photography. However, the present invention isnot restricted to the present embodiment. It is acceptable that theimage taking system of the present invention is, for example, one havinga high-speed type of flashlight emitting device in which in the eventthat it is permitted to take a very short time for charge and thephotography is carried out continuously twice, it is allowed to emit theflashlight twice. It is noted, however, that such an image taking systemalso needs the intercept circuit for intercepting the unnecessaryinstruction signal.

1. A background replacing apparatus comprising: an image obtainingsection that obtains a plurality of photographic images generatedthrough photography of a common subject under a plurality ofphotographic conditions; an area discriminating section thatdiscriminates among a subject area, a background area, and a boundaryarea of the photographic image in accordance with at least one of thephotographic images obtained by the image obtaining section; a mixingratio determining section that determines a mixing ratio of a subjectcolor to a background color in the boundary area; and a backgroundreplacing section that replaces a background of the photographic imageobtained by the image obtaining section with a different background inaccordance with three areas of the subject area, the background area,and the boundary area, which are discriminated by the areadiscriminating section, and the mixing ratio determined by the mixingratio determining section.
 2. A background replacing apparatus accordingto claim 1, wherein the area discriminating section discriminates amongthe subject area, the background area, and the boundary area of thephotographic image in accordance with a predetermined different pointbetween the subject and the background in the photographic conditions.3. A background replacing apparatus according to claim 1, wherein thearea discriminating section discriminates the photographic image betweena subject-side area and a background-side area in accordance with apredetermined different point between the subject and the background inthe photographic conditions, and at least one of the subject-side areaand the background-side area is reduced so as to be treated as thesubject area and the background area, and an area between the subjectarea and the background area is treated as the boundary area.
 4. Abackground replacing apparatus according to claim 1, wherein the areadiscriminating section discriminates the photographic image between asubject-side area and a background-side area in accordance with apredetermined different point between the subject and the background inthe photographic conditions, and the subject-side area is reduced so asto be treated as the subject area, and an area portion excepting thesubject area, of the subject-side area before reduction, is treated aswhole or part of the boundary area.
 5. A background replacing apparatusaccording to claim 1, wherein the area discriminating sectiondiscriminates the photographic image between a subject-side area and abackground-side area in accordance with a predetermined different pointbetween the subject and the background in the photographic conditions,and the background-side area is reduced so as to be treated as thebackground area, and an area portion excepting the background area, ofthe background-side area before reduction, is treated as whole or partof the boundary area.
 6. A background replacing apparatus according toclaim 1, wherein the area discriminating section discriminates among thesubject area, the background area, and the boundary area of thephotographic image in accordance with one of the photographic imagesobtained by the image obtaining section, and the background replacingsection replaces a background of a photographic image, which isdifferent from the photographic image used for discrimination of areasby the area discriminating section, of the photographic images obtainedby the image obtaining section.
 7. A background replacing apparatusaccording to claim 1, wherein the mixing ratio determining sectionpresumes the subject color and the mixing ratio in the boundary area ofthe photographic image; and the background replacing section replacesthe background of the photographic image with the different backgroundusing the subject color and the mixing ratio, which are presumed by themixing ratio determining section.
 8. A background replacing apparatusaccording to claim 7, wherein the mixing ratio determining sectionpresumes the subject color using colors in the subject area ascandidates for the subject color.
 9. A background replacing apparatusaccording to claim 7, wherein the mixing ratio determining sectionpresumes the subject color and the mixing ratio on individual points inthe boundary area, and presumes the subject color on assumption that thesubject color lines up on a straight line basis with the backgroundcolor and a color of the associated point in a predetermined colorspace.
 10. A background replacing apparatus according to claim 7,wherein the mixing ratio determining section presumes the subject colorand the mixing ratio on individual points in the boundary area, andpresumes the subject color, using colors in the subject area ascandidates for the subject color, in such a manner that of thecandidates, a candidate, which lines up on a straight line basis withthe background color and a color of the associated point in apredetermined color space, and is closest to the color of the associatedpoint, is presumed as the subject color.
 11. A background replacingapparatus according to claim 7, wherein the mixing ratio determiningsection presumes the subject color and the mixing ratio on individualpoints in the boundary area, and presumes the mixing ratio in accordancewith a ratio of a mutual distance among the background color, a color ofthe associated point, and the subject color, in a predetermined colorspace.
 12. A background replacing apparatus according to claim 7,wherein the background replacing section replaces the background of thephotographic image with the different background in such a manner that acolor of the different background and the subject color presumed by themixing ratio determining section are mixed at the mixing ratio presumedby the mixing ratio determining section.
 13. A background replacingapparatus according to claim 1, wherein the background replacingapparatus further comprises an image correcting section performscorrection of the area discriminated by the area discriminating sectionin response to an operation, and causes the background replacing sectionto perform the background replacing based on a corrected area so that abackground replaced image is corrected.
 14. A background replacingapparatus according to claim 13, wherein the area discriminating sectiondiscriminates the photographic image between a subject-side area and abackground-side area in accordance with a predetermined different pointbetween the subject and the background in the photographic conditions,and the subject-side area and the background-side area are reduced by apredetermined reduction amount so as to be treated as the subject areaand the background area, and the image correcting section alters thereduction amount on the area discriminating section in response to anoperation, so that the subject area and/or the background area arecorrected.
 15. A background replacing apparatus according to claim 13,wherein the area discriminating section discriminates between thesubject area and the background area in accordance with a discriminationbasis based on a predetermined different point between the subject andthe background in the photographic conditions, and the image correctingsection alters the discrimination basis in response to an operation, sothat the subject area and/or the background area are corrected.
 16. Abackground replacing apparatus comprising: an image obtaining sectionthat obtains a plurality of photographic images generated throughphotography of a common subject under a plurality of photographicconditions; an area discriminating section that discriminates abackground area of the photographic image from other areas in accordancewith at least one of the photographic images obtained by the imageobtaining section; a mixing state presuming section that presumes amixing ratio of a subject color and a background color in other areasexcepting the background area in the photographic image, and the subjectcolor; and a background replacing section that replaces a background ofthe photographic image obtained by the image obtaining section with adifferent background using the subject color and the mixing ratio, whichare presumed by the mixing state presuming section.
 17. A backgroundreplacing apparatus according to claim 16, wherein the mixing statepresuming section presumes the subject color using colors in the subjectarea as candidates for the subject color.
 18. A background replacingapparatus according to claim 16, wherein the mixing state presumingsection presumes the subject color and the mixing ratio on individualpoints in other areas excepting the background area, and presumes thesubject color on assumption that the subject color lines up on astraight line basis with the background color and a color of theassociated point in a predetermined color space.
 19. A backgroundreplacing apparatus according to claim 16, wherein the mixing statepresuming section presumes the subject color and the mixing ratio onindividual points in other areas excepting the background area, andpresumes the subject color, using colors in other areas excepting thebackground area as candidates for the subject color, in such a mannerthat of the candidates, a candidate, which lines up on a straight linebasis with the background color and a color of the associated point in apredetermined color space, and is farthest from the color of theassociated point, is presumed as the subject color.
 20. A backgroundreplacing apparatus according to claim 16, wherein the mixing statepresuming section presumes the subject color and the mixing ratio onindividual points in other areas excepting the background area, andpresumes the mixing ratio in accordance with a ratio of a mutualdistance among the background color, a color of the associated point,and the subject color, in a predetermined color space.
 21. A backgroundreplacing apparatus according to claim 16, wherein the backgroundreplacing section replaces the background of the photographic image withthe different background in such a manner that a color of the differentbackground and the subject color presumed by the mixing state presumingsection are mixed at the mixing ratio presumed by the mixing statepresuming section.
 22. A background replacing apparatus according toclaim 16, wherein the area discriminating section discriminates abackground area of the photographic image from other areas in accordancewith a ratio of luminous intensity among the plural photographic images.23. A background replacing apparatus according to claim 16, furthercomprising a storage section that stores the subject color and themixing ratio, which are presumed by the mixing state presuming section,wherein the background replacing section replaces a background of thephotographic image obtained by the image obtaining section with adifferent background using the subject color and the mixing ratio, whichare stored in the storage section.
 24. A background replacing apparatusaccording to claim 16, further comprising a first parameter determiningsection that determines a correction parameter to be used for a colorcorrection for the subject color in accordance with an image of saidother area in which a color is replaced by the subject color presumed inthe mixing state presuming section, and a first color correcting sectionthat applies the color correction to the subject color, which ispresumed by the mixing state presuming section, using the correctionparameter determined by the first parameter determining section, whereinthe background replacing section replaces the background of thephotographic image obtained by the image obtaining section with adifferent background using the subject color subjected to the colorcorrection by the first color correcting section and the mixing ratio.25. A background replacing apparatus according to claim 16, furthercomprising a second parameter determining section that determines acorrection parameter to be used for a color correction for the subjectcolor in accordance with the photographic image obtained by the imageobtaining section, and a second color correcting section that appliesthe color correction to the subject color, which is presumed by themixing state presuming section, using the correction parameterdetermined by the second parameter determining section, wherein thebackground replacing section replaces the background of the photographicimage obtained by the image obtaining section with a differentbackground using the subject color subjected to the color correction bythe second color correcting section and the mixing ratio.
 26. Abackground replacing apparatus according to claim 16, further comprisinga third parameter determining section that determines a correctionparameter to be used for a color correction for the subject color to aparameter according to an operation, and a third color correctingsection that applies the color correction to the subject color, which ispresumed by the mixing state presuming section, using the correctionparameter determined by the third parameter determining section, whereinthe background replacing section replaces the background of thephotographic image obtained by the image obtaining section with adifferent background using the subject color subjected to the colorcorrection by the third color correcting section and the mixing ratio.27. A background replacing apparatus according to claim 16, furthercomprising a fourth parameter determining section that determines acorrection parameter to be used for a color correction for the subjectcolor to a parameter according to the different background, and a fourthcolor correcting section that applies the color correction to thesubject color, which is presumed by the mixing state presuming section,using the correction parameter determined by the fourth parameterdetermining section, wherein the background replacing section replacesthe background of the photographic image obtained by the image obtainingsection with the different background using the subject color subjectedto the color correction by the fourth color correcting section and themixing ratio.
 28. A background replacing apparatus according to claim16, wherein the background replacing section replaces a background ofthe photographic image using, as the different background, a backgroundto which relation information representative of a relative positionalrelation in an image between the subject and the background is applied,and performs an replacement in such a manner that the relative relationin the background replaced image between the subject and the backgroundis the same relative relation as the relative relation represented bythe relation information.
 29. A background replacing apparatuscomprising: an image obtaining section that obtains a plurality ofphotographic images generated through photography of a common subjectunder a plurality of photographic conditions mutually different in colorof the background; an area discriminating section that discriminates abackground area of the photographic image from other areas in accordancewith the photographic images obtained by the image obtaining section; amixing state presuming section that presumes a mixing ratio of thesubject color to the background color, and the subject color, in otherareas excepting the background area in the photographic image, inaccordance with the plurality of photographic images; and a backgroundreplacing section that replaces a background of the photographic imageobtained by the image obtaining section with a different backgroundusing the subject color and the mixing ratio, which are presumed by themixing state presuming section.
 30. A background replacing apparatusaccording to claim 29, wherein the mixing state presuming sectionpresumes, on assumption that the plurality of photographic images areequal to each other in terms of mixing state of the subject color andthe background color, the subject color and the mixing ratio.
 31. Abackground replacing apparatus according to claim 19, wherein the mixingstate presuming section presumes the subject color and the mixing ratioon individual points in other area excepting the background area,determines straight lines each coupling in a predetermined color spacethe background color with a color of the associated point in each of theplurality of photographic images generated through photography of thecommon subject under a plurality of photographic conditions mutuallydifferent in color of the background, and presumes a color correspondingto an intersection of the straight lines as the subject color.
 32. Abackground replacing apparatus according to claim 29, wherein the imageobtaining section obtains a plurality of photographic images generatedthrough photography of the common subject under a plurality ofphotographic conditions mutually different in color-phase of thebackground.
 33. A background replacing apparatus according to claim 29,wherein the area discriminating section discriminates between a subjectarea and a boundary area of other areas excepting the background area inthe photographic image; and the mixing state presuming section thatpresumes the subject color and the mixing ratio in the boundary area.34. A background replacing apparatus according to claim 29, wherein thearea discriminating section discriminates the background area of thephotographic image obtained by the image obtaining section from otherareas in accordance with an amount of the change in color among theplurality of photographic images.
 35. A background replacing apparatusaccording to claim 29, wherein the area discriminating sectiondiscriminates the background area of the photographic image obtained bythe image obtaining section from other areas in accordance with adifference of luminous intensity between the background and the subject.36. A background replacing program that causes a computer to operate asan apparatus, when the background replacing program is incorporated intothe computer and is executed, the apparatus comprising: an imageobtaining section that obtains a plurality of photographic imagesgenerated through photography of a common subject under a plurality ofphotographic conditions; an area discriminating section thatdiscriminates among a subject area, a background area, and a boundaryarea of the photographic image in accordance with the photographicimages obtained by the image obtaining section; a mixing ratiodetermining section that determines a mixing ratio of a subject color toa background color in the boundary area; and a background replacingsection that replaces a background of the photographic image obtained bythe image obtaining section with a different background in accordancewith three areas of the subject area, the background area, and theboundary area, which are discriminated by the area discriminatingsection, and the mixing ratio determined by the mixing ratio determiningsection.
 37. A background replacing program that causes a computer tooperate as an apparatus, when the background replacing program isincorporated into the computer and is executed, the apparatuscomprising: an image obtaining section that obtains a plurality ofphotographic images generated through photography of a common subjectunder a plurality of photographic conditions; an area discriminatingsection that discriminates a background area of the photographic imagefrom other areas in accordance with the photographic images obtained bythe image obtaining section; a mixing state presuming section thatpresumes a mixing ratio of the subject color and the background color,and the subject color, in other areas excepting the background area inthe photographic image; and a background replacing section that replacesa background of the photographic image obtained by the image obtainingsection with a different background using the subject color and themixing ratio, which are presumed by the mixing state presuming section.38. A background replacing program that causes a computer to operate asan apparatus, when the background replacing program is incorporated intothe computer and is executed, the apparatus comprising: an imageobtaining section that obtains a plurality of photographic imagesgenerated through photography of a common subject under a plurality ofphotographic conditions in which colors of backgrounds are differentfrom one another; an area discriminating section that discriminates abackground area of the photographic image from other areas in accordancewith the photographic images obtained by the image obtaining section; amixing state presuming section that presumes a mixing ratio of thesubject color and the background color, and the subject color, in otherareas excepting the background area in the photographic image, inaccordance with the plurality of photographic images; and a backgroundreplacing section that replaces a background of the photographic imageobtained by the image obtaining section with a different backgroundusing the subject color and the mixing ratio, which are presumed by themixing state presuming section.
 39. A background replacing methodcomprising: an image obtaining step that obtains a plurality ofphotographic images generated through photography of a common subjectunder a plurality of photographic conditions; an area discriminatingstep that discriminates among a subject area, a background area, and aboundary area of the photographic image in accordance with thephotographic images obtained by the image obtaining step; a mixing ratiodetermining step that determines a mixing ratio of a subject color to abackground color in the boundary area; and a background replacing stepthat replaces a background of the photographic image obtained by theimage obtaining step with a different background in accordance withthree areas of the subject area, the background area, and the boundaryarea, which are discriminated by the area discriminating step, and themixing ratio determined by the mixing ratio determining step.
 40. Abackground replacing method comprising: an image obtaining step thatobtains a plurality of photographic images generated through photographyof a common subject under a plurality of photographic conditions; anarea discriminating step that discriminates a background area of thephotographic image from other areas in accordance with the photographicimages obtained by the image obtaining step; a mixing state presumingstep that presumes a mixing ratio of a subject color and a backgroundcolor, and the subject color, in other areas excepting the backgroundarea in the photographic image; and a background replacing step thatreplaces a background of the photographic image obtained by the imageobtaining step with a different background using the subject color andthe mixing ratio, which are presumed by the mixing state presuming step.41. A background replacing method comprising: an image obtaining stepthat obtains a plurality of photographic images generated throughphotography of a common subject under a plurality of photographicconditions; an area discriminating step that distinguishes a backgroundarea of the photographic image from other areas in accordance with thephotographic images obtained by the image obtaining step; a mixing statepresuming step that presumes a mixing ratio of a subject color and abackground color, and the subject color, in other areas excepting thebackground area in the photographic image, in accordance with theplurality of photographic images; and a background replacing step thatreplaces a background of the photographic image obtained by the imageobtaining step with a different background using the subject color andthe mixing ratio, which are presumed by the mixing state presuming step.